NILPOTENT GELFAND PAIRS 
AND SPHERICAL TRANSFORMS OF SCHWARTZ FUNCTIONS 
III. ISOMORPHISMS BETWEEN SCHWARTZ SPACES 
UNDER VINBERG'S CONDITION 

VERONIQUE FISCHER, FULVIO RICCI, OKSANA YAKIMOVA 

Abstract. Let(N, K) be a nilpotent Gelfand pair, i.e., N is a nilpotent Lie group, K a 
compact group of automorphisms of N, and the algebra D(N) K of left-invariant and K- 
invariant differential operators on N is commutative. In these hypotheses, N is necessarily 
of step at most two. We say that (N, K) satisfies Vinberg's condition if K acts irreducibly 
on n/[n,n], where n = Lie(iV). 

Fixing a system D of d formally self-adjoint generators of O(iV) , the Gelfand spectrum 
of the commutative convolution algebra L 1 (^V) if can be canonically identified with a closed 
subset Xx> of E d . We prove that, on a nilpotent Gelfand pair satisfying Vinberg's condition, 
the spherical transform Q : L 1 (N) K i — > Co(S-p) establishes an isomorphism from the space 
S(N) K of if-invariant Schwartz functions on N and the space S(Y,x>) of restrictions to £d 
of functions in <S(M d ). 
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1. Introduction 

Let N be a connected, simply connected nilpotent Lie group and K a compact group of 
automorphisms of N. We say that (N,K) is a nilpotent Gelfand pair (n.G.p. in short )Q if 
either of the following equivalent condition is satisfied: 

(i) the convolution algebra L l (N) K of i^-invariant integrable functions on is commu- 
tative; 

(ii) the algebra D(A^)^ of left-invariant and i^-invariant differential operators on A^ is 
commutative. 

According to the common terminology, this is the same as saying that (K x N, K) is a 
Gelfand pair. The expression "commutative nilmanifold" is used for (K x N) / K in [26] . 

The relevance of nilpotent Gelfand pairs in the class of general Gelfand pairs is emphasized 
by Vinberg's structure theorem [221 Th. 5]. 

According to the Gelfand theory of commutative Banach algebras, harmonic analysis on 
Gelfand pairs is based on the notions of spherical function and spherical transform [B], [H] 
Ch. IV]. For nilpotent pairs, spherical functions can be defined as the joint iT-invariant 
eigenfunctions <p of all operators in D(A^)^ which take value 1 at the identity. The spherical 
transform of a function F G L 1 (N) K is 

(1.1) QF(ip)= [ F(x) V (x~ l ) dx , 

J N 



When dealing with specific pairs, we will find it convenient to identify N with its Lie algebra n and write 
(n,K) instead of (N,K). 
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defined on the Gelfand spectrum of the pair, E = S(iV, K), i.e., the space of bounded spherical 
functions endowed with the compact-open topology. Then 

G : L\N) K — ► C (E) 

and is continuous. 

The Gelfand spectrum E admits natural embedding in Euclidean spaces. Let 

V= (D h ...,D d ) , 

be a c/-tuple of essentially self-adjoint operators which generate V>(N) K as an algebra. Every 
bounded spherical function ip is identified by the d-tuple £ = £(</?) = • • • , ^(v 9 )) °f 

eigenvalues of ^ relative to Di, . . . , respectively. The d-tuples £((p) form a closed subset 



Ed of R which is homeomorphic to E [7j. Hence the spherical transform QF in (1.1) can 
be viewed function on Ed. 

In the case where iV = IR n and is trivial, D(M n )' R " is the algebra of all constant coefficient 
differential operators, and the bounded spherical functions are the unitary characters <fi\(x) = 
e iX ' x , for A G E n . Taking 

V= (i~ l d xl ,...,i~ l d Xn ) , 

we have £((p\) = A, so that Ed = R n , and QF = F is the ordinary Fourier transform. 

It has been conjectured in [S] and [9], that the invariance under Fourier transform of the 
Schwartz space 5(lR n ), a fundamental fact in Fourier analysis, has an analogue on nilpotent 
Gelfand pairs, in the sense that the spherical transform gives a bijective correspondence 
between i^-invariant Schwartz functions on N and restrictions to Ed of Schwartz functions 
on R d . 

To make the statement precise, denote by S(N) K the space of K-invariant Schwartz 
function on N and by 

5(ED)=5(M d )/{/:/| Sl , = 0} 

the space of restrictions to Ed of Schwartz functions on R d , with the quotient topology. The 
conjectured property, for any n.G.p. (N,K), is as follows: 

(S) The spherical transform Q maps the space S(N) K isomorphically onto S (£,%>). 

The problem is well posed because the answer does not depend on the choice of T> [2] 
and 0. 

Property (S) has been proved to hold in several cases. For "abelian pairs", i.e., with 
N = W 1 and K C GL n (R) compact, it has been shown in [2] that Property (S) follows from 
G. Schwarz's extension [19] of Whitney's theorem [25] to general linear actions of compact 
groups on W n . 

For nonabelian N, Property (S) has been proved in the following cases: 

(i) pairs in which iV is a Heisenberg group or a complexified Heisenberg group [H [2]; 

(ii) the pair (H n © ImH, SpJ pQ; 

(iii) "rank-one" pairs, where [n, n] = 3, the centre of n, and the if-orbits in 3 are full 
spheres [HI E] • 
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Part of the statement is a matter of functional calculus on Rockland operators on graded 
groups (i.e., with a graded Lie algebra). It was proved in [15] that if L is a Rockland operator 
and g is a Schwartz function on the line, then the operator g(L) is given by convolution with 
a Schwartz kernel. This statement has been later extended to commuting families of d 
Rockland operators and g G in [21] in a special case, and in [21 Th. 5.2] in general. 

Since on any n.G.p. we always have a system D consisting of Rockland operators [2] and 

g(D 1 ,...,D d )f = f*K GK = g lsv , 

this has the following consequence. 

Theorem 1.1 ([21 |Hj). Let (N,K) be a nilpotent Gelfand pair, and T>, C M. d as above. 
Given any Schwartz function g on WL d , there is a K -invariant Schwartz function F on N , 
depending continuously on m, such that QF = g\ s ^ . 

In other words, we always have the continuous inclusion 

g(S(N) K ) D5(E„) . 

So the proof of Property (S) reduces to proving the opposite inclusion, i.e., that the 
spherical transform of any function in S(N) K admits a Schwartz extension to M. d . 

For all pairs with nonabelian N studied in [H [21 El Ellin], the proof contains a bootstrapping 
argument (see below), which makes the validity of Property (S) at a given stage a necessary 
requirement for proving it in more complex or general situations. In a more systematic way, 
the results of this paper are proved via an inductive procedure, which involves a similar 
bootstrapping, and also relies on the validity of Property (S) in the above mentioned cases 
(i)-(iii). 

It turns out very useful in this process to have at hand a classification of all nilpotent 
Gelfand pairs. Their knowledge makes it possible to develop a general strategy of proof and 
simplify the most technical parts. 

A first classification of nilpotent Gelfand pairs was obtained by E. Vinberg in [2"2l [23] , 
under the following assumption, which we call Vinberg 's condition: 

(V) K acts irreducibly on n/[n, n]. 

The list of all pairs satisfying Vinberg's condition is in [22, Table 3] (with an inaccu- 
racy corrected in [2H])- The classification of all nilpotent Gelfand pairs was completed by 
O. Yakimova (23121], see also [211 Ch. 13, 15]. 

Vinberg's list can also be found in the Appendix of [9], where families of fundamental 
invariants are obtained for each case. 

We can state now our main theorem. 

Theorem 1.2. Property (S) holds for all nilpotent Gelfand pairs satisfying condition (V). 

In the course of the proof, it will be sufficient to limit our analysis to pairs in a much more 
restricted list. In fact, we can disregard the pairs in (i)-(iii) above, as well as many others 
according to the following principles: 

(a) if Property (S) holds for (N, K), and K is normal in a larger compact group K# of 
automorphisms of N, then it also holds for (N,K#) (normal extension); 
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(b) if Property (S) holds for (N,K), and the centre 3 of n has a nontrivial proper K- 
invariant subspace s, then it also holds for (N/ exps,K) (central reduction). 

The resulting reduced list of pairs to work on is given in Table [Tj 

We also mention here a third principle which will be used in the course of the proof: 

(c) if Property (S) holds for two n.G.p., (Ni,Kx) and (iV^-fG), it also holds for the 
product pair (Ni x N2,Ki x K2). 

The proof of (a) is contained in [9], and (b), (c) will be proved in Section 13} 



In order to present the main ideas in this paper, we first sketch the scheme used in the 
previous papers [H 121 El El EH] on this subject. 

Inside one can identify an open "regular" set and a complementary "singular" set. 
The regular set has the property that any point £ in it has a neighbourhood which 
is homeomorphic to an open neighbourhood V v of some point in the Gelfand spectrum of 
another n.G.p., which is "simpler" in the sense that it is already known to satisfy Property 
(S). More precisely, there is a natural homeomorphism which induces, by composition, a 
correspondence between C°°-functions on and C°°-functions on V v . Using a partition of 
unity, this argument is sufficient to imply the existence of Schwartz extensions to M. d when 
the spherical transform QF of a function F G S(N) K vanishes of infinite order on the singular 
set. This is the "bootstrapping" part of the argument. 

Such reduction to simpler pairs is not possible on the singular set. However, the singular 
set itself is identified with the Gelfand spectrum of a n.G.p. for which Property (S) is known 
to hold. At this point, a Hadamard-type formula for K- invariant smooth functions on N 
produces, for any given F G S(N) K , a Whitney jet of infinite order on the singular set, i.e., 
it determines the derivatives that any smooth extension of QF must have on the singular set. 
By Whitney extension theorem [23], there is a Sch wart z function g on R d with the prescribed 
derivatives on the singular set. Applying Theorem we find a function G G S(N) K having 
g as its spherical transform. Since the difference QiF — G) vanishes of infinite order on the 
singular set, the bootstrapping argument allows to conclude the proof. 

One novelty of this paper is that the notions of regular and singular set must be refined by 
taking into account the "higher-rank" nature of the action of K on the centre 3 of n, which 
produces different levels of singularity of points in E75. 

Taking into account that in every n.G.p. the group N has step at most two [3], and that 
[n, n] = 3 for all pairs in Vinberg's list, we can associate to each bounded spherical function, 
i.e., to each point in the Gelfand spectrum, a conjugacy class, modulo the action of K, of 



irreducible unitary representations of N, and hence a f^-orbit in 3* = 3 (cf. Section 2.1). 

Let (p be a bounded spherical function and t any point in the corresponding orbit O v C 3. 
Denoting by N t the quotient group of iV with Lie algebra n/T t O v and by K t the stabilizer 
of t in K, p projects to N t as a spherical function p 1 for the n.G.p. (N t , K t ). 

The quotient pair (N t ,K t ) of (N,K) measures the level of singularity of ip, or of £(<p) as 
a point of Sp. The highest level of singularity occurs when t is fixed by K. In this case 
(N t , Kt) = (N, K). In all other cases, (N t , K t ) is a proper quotient pair. 

For instance, consider the pairs at line 2 of Table [TJ where 

• n = C n © u n with Lie bracket [(v, z), (v ', z')] = (0, vv'* - v'v*) , 
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• K = U n acts on n by k ■ (v, z) = (kv, kzk*). 

Then the action of K on 3 = u n is the adjoint action. In this case, our notion of singularity 
of a spherical function matches with the notion of singularity of a point in a Cartan subal- 
gebra of u n , with the level of singularity measured by the set of positive roots annihilating 
it. In this example, if the orbit associated to a given spherical function contains the element 
t = diag(ti/ pi , . . . ,t k I Pk ) with different tj's, the quotient pair (N t ,K t ) is a product of pairs 
(C Pj ©u Pj , Xl Pj ) (cf. Section 10.2). This is a proper quotient pair unless t is a scalar multiple 



of the identity matrix. 

In Section [5| we prove that the map tp 1 — > y?* gives a local homeomorphism of a neigh- 
bourhood of if in E onto a neighbourhood of tp 1 in the Gelfand spectrum E* of (N t ,K t ). 
The proof is based on the existence of slices transversal to i^-orbits in 3, which allows local 
extensions near t of smooth .fQ-invariant functions on N t to smooth if-invariant functions 
on N (radialisation). In Section [6j we prove that, given two realisations Ed, E|> t of the two 
spectra in Euclidean spaces, this homeomorphism induces a local identification, near the 
points corresponding to ip and (p f respectively, of the two spaces <S(Ed) and <S(E Ct ). 

This result allows to prove the following version of the bootstrapping argument, cf. Sec- 
tion [7} Denote by Ed the set of "most singular points" in Ed, i.e., those for which 
(Nt, K t ) = (N, K), and assume that Property (S) holds for all proper quotient pairs of (N, K). 
If the spherical transform QF of a function F G S(N) K vanishes of infinite order on Ed, 
then it can be extended to a Schwartz function on M. d . 

Let us assume for a moment that all proper quotient pairs of (N, K) satisfy Property (S). 
Following the general pattern, we show in Section [8] that QF determines a Whitney jet on 
Ed- This is a consequence of two facts. 

The first fact is that Ed is naturally identified with the Gelfand spectrum of (N, K), where 
the Lie algebra of N is n/30, where 30 is the component of 3 on which K acts nontrivially, 



cf. (2.1). More precisely, for an appropriate choice of the system T> of generators of O(iV) 



K 



we construct a system T> of d' < d generators of 3(N) K such that 

t v = Ed n (R d ' x {0}) = Ed x {0} . 



The second fact is the Hadamard-type formula of Proposition 8.2 Once read on the other 
side of the spherical transform, it has the form of the inductive step for a Taylor development 
of QF, for F G S(N) K , centred on R d ' x {0}, in the remaining d — d' variables. This provides 
the desired Whitney jet, once we observe, by case by case inspection, that (N, K) is one of 
the pairs for which Property (S) is already known to hold (e.g., in the above example, N is 
a Heisenberg group) . 



The proof of Proposition 8.2 relies on the preliminary Proposition 8.1, which has already 



appeared in [10]. In this paper we only show how Proposition 8.1 implies Proposition 8.2 



We remark that the proof of Proposition 8.1 given in [10] is based on the fact that certain 



tensor products of irreducible representations of K decompose without multiplicities, and 
this requires a case by case analysis. 

It remains to answer the question if Property (S) is satisfied by all proper quotient pairs 
(N t , Kt) generated by a pair (N, K) in Table [!} The list of such quotient pairs is given in 
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Section [TUJ One can notice that the pairs in the first two blocks of Table [T] form a self- 
contained family, in the sense that the quotient pairs that they generate are products of 
pairs in the same family with lower dimensional groups. This allows an inductive argument 
using principle (c) above. 

The situation is different for the pairs in the third block, since the quotient pairs they 
generate do not satisfy condition (V) . For them we must provide an ad-hoc adaptation of the 
previous argument, with the inductive procedure replaced by analysis of three consecutive 
generations of quotient pairs. This is done in Section [9| We are confident that this extra 
work on isolated cases will turn out to be useful in view of a future proof of Property (S) for 
general pairs not satisfying Vinberg's condition. 

Acknowledgments. This work started when the third author was a long-term visitor at 
Centro di Ricerca Matematica Ennio de Giorgi in Pisa. Part of it was later carried out at 
the Max-Planck-Institute fiir Mathematik (Bonn), where all the three authors have been 
short-term guests. We would like to thank these institutes for their warm hospitality and 
for providing an excellent stimulating environment. 

The first author acknowledges the support of Scuola Normale Superiore, Pisa, and of the 
London Mathematical Society Grace Chisholm fellowship held at King's College, London. 
The third author also wishes to thank Scuola Normale Superiore for regular invitations and 
D. Timashev for bringing the book of Bredon to her attention. 

2. Generalities on nilpotent Gelfand pairs 



Let N be a nilpotent, connected and simply connected Lie group, and let K be a compact 
group of automorphisms of N. 

Definition 2.1. (N, K) is a nilpotent Gelfand pair (n. G.p. in short) if either of the following 
equivalent conditions is satisfied: 

(i) the convolution algebra L 1 (N) K of integrable K -invariant functions on N is commu- 
tative; 

(ii) the algebra 3(N) K of left-invariant and K -invariant differential operators on N is 
commutative; 

(iii) if ii is an irreducible unitary representation of N and K.,, is the stabilizer in K of 
the equivalence class of tt, then the representation space "H^ decomposes under K n 
without multiplicities; 

(iv) same as (iii) ; for n generic. 

This is the same as saying that (K x N, K) is a Gelfand pair. With n denoting the Lie 
algebra of N, we often write (n, K) instead of (N, K). 

In any nilpotent Gelfand pair, N has step at most 2 [3] . We can then split n as the direct 
sum t) © [n, n], where [n, n] is the derived algebra and t) a K- invariant complement of it. For 
pairs satisfying Vinberg's condition, the derived algebra coincides with the centre 3, and it 
will henceforth be denoted by this symbol. 
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We regard the Lie bracket on n as a skew-symmetric bilinear map from D x D to 3. We 
split 3 as 



where 3 denotes the subspace of K- fixed elements of 3, and 30 its (unique) if-invariant 
complement in 3. 

We constantly use exponential coordinates on iV to identify elements of iV with elements 
of n. In particular, the product on N is expressed as an operation on D © 3, via the Baker- 
Campbell-Hausdorff formula 



Let A = \n be the standard symmetrisation operator from the symmetric algebra 6(n) 
onto the universal enveloping algebra il(n), which is linear and satisfies the identity X(X n ) = 
X n for every l6n and n G N. As usual, we regard <5(n) as the space V(n*) of polynomials 
on the dual space n*. When the elements of ll(tl) are regarded as left-invariant differential 
operators on N, we use the notation O(iV). 

Following [9 S Sect. 2.2], we will use a modified symmetrisation X' N : V{xC) — > D(iV), 
which maps the polynomial p G P(n) to the differential operator 



specify the group N, we write A^y instead of A'. The advantage of this modification is that 
polynomials with real coefficients are transformed by A' into formally self-adjoint differential 
operators^} Clearly, \'(p) is if-invariant if and only if p is if- invariant. 

Introducing a if-invariant inner product ( , ) on t) © 3 under which _L 3, we identify n* 
with n throughout the paper. 

2.1. Spherical functions, representations of iV and K-orbits in 3. 

Let (N,K) be a n.G.p. The spherical functions are the if-invariant joint eigenfunctions 
ip of all operators in 3(N) K normalised by the condition y?(0) = 1. Given D G 3(N) K and 
a spherical function ip, we denote by £,(D,cp) the corresponding eigenvalue. 

We are interested in the bounded spherical functions, which play the main role in the 
Fourier- Go dement analysis of Gelfand pairs. 

It has been proved in [3] that all bounded spherical functions of (N, K) are of positive type, 
hence they are in one-to-one correspondence with (equivalence classes of) irreducible unitary 
representations of K x iV admitting non-trivial if-invariant vectors. However, we prefer to 
avoid representations of the semidirect product and express bounded spherical functions as 
partial traces of irreducible unitary representations of N. 

For ( G 3, denote by C d the radical of the bilinear form B^(v,v') = (£, and 
set t)£ = tf. The following statement is a direct consequence of the Stone- von Neumann 
theorem and we omit its proof. 

2 The first two authors take this opportunity to correct an error in the formulation of Proposition 3.1 in 
[5]: it applies to operators D = X'(p) with p real. 



(2.1) 



3o ©3 , 
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Lemma 2.2. For each ( £ 3 t/iere is a unique, up to equivalence, irreducible unitary rep- 
resentation 7T£ of N such that dii^(0,z) = i((,z)I for all z £ 3 and dn^(v,0) = for all 

V £ . 

For eac/i £ £ 3 and a; £ t£ t/iere is a unique irreducible unitary representation such 
that 

(2.3) dir£ t u(v, z) = dn^(v, z) + i(v, u)I . 

Every irreducible unitary representation of N is equivalent to one, and only one, iv^ u . 

We denote by the representation space of the representations ir^. The stabilizer 
Kq j0J C K of the point uj + ( £ n also stabilizes the equivalence class of ir^u, inducing 
a unitary representation^] a of on "H^. By [51 [22], the fact that (N,K) is a n.G.p. is 
equivalent to saying that, for each £, a;, decomposes without multiplicities into irreducible 
components under the action of K^ u , namely, 

(2.4) H C = J2 > 

with 3£f )W C K^ jU . To each p £ X^ ja; we can associate the spherical function 

(2.5) z ) = dim v^ J tr ( n <A kv i kz )\ vi J dk ■ 

For given k £ K, we have 3Lkc,ku = 3tc>^j under the natural identification of the dual object 
Kfu of i<V ui with the dual object of K k = k~ l K n k, and 

Si Si ' c,,w 



2.2. Spectra and their immersions in K . 

Given a n.G.p. (N,K), we denote by S, or T±(N,K), the Gelfand spectrum of L 1 (N) K , 
i.e. the set of bounded .fT-spherical functions on N with the compact-open topology. 

By a homogeneous Hilbert basis, or a fundamental system of invariants, we mean a d- 
tuple p = (pi, . . . , pa) of real, i^-invariant polynomials on n which generate the .fT-invariant 
polynomial algebra V(n) K over n and with each pj homogeneous in the D-variables and in 
the 3-variables separately, i.e., belonging to V rj (t>) <8> V Sj {])) for some rj, Sj. 

We set Dj = A'(Pj) and V = (D 1 ,...,D d ). Then V generates B(N) K . We call it a 
homogeneous basis of 3(N) K . 

By Proposition 3.1 of [5], the Dj are essentially self-adjoint on S(N) and their closures 
admit a joint spectral resolution. 

Given <p £ S, denote by £j(<p) £ K the eigenvalue £(Dj, <p) of <p under Dj. Then 

(2.6) S„ = {t{ V ) = (^(cp), • • • , : up £ S} C R d 

3 In general, this operation leads to a projective representation of the stabilizer in K . In our case we obtain 
true representations, since restriction of the metaplectic representation of Sp(r^, Bq) to a compact subgroup 
can be linearized [IT] . 
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is closed and homeomorphic to S, cf. [7]. Moreover, is the joint L 2 -spectrum of D as a 
family of strongly commuting self-adjoint operators [8]. 

If p = (pi, . . . , pg) is another real, homogeneous Hilbert basis of V{n) K and V is the 
corresponding homogeneous basis of D(iV) , there are polynomials Pk, Pj such that 

(2.7) D k = Pfcpi, • • • , D d ) , £>, = P.pi, • ■ • , Di) , 

for 1 < j < d, 1 < k < d. Hence = P(S C ) and S c = P(E^). 

The following statement is proved in [2] (cf. Lemma 3.1 and Corollary 3.2 therein). 

Proposition 2.3. The validity of Property (S) is independent of the choice of p (i.e. ofT>). 

On N, as well as on its Lie algebra, we consider the automorphic dilations 5 ■ (v, z) = 
(5zv,5z), defined for 5 > 0. If p G V(n) is homogeneous of degree v' in the D-variables and 
of degree v" in the 3- variables, i.e., p € V v (t>) <8> V v (3), then the operator D = X'(p) is 
homogeneous of degree v — |- +u" with respect to the automorphic dilations, i.e., it satisfies 
the identity 

(2.8) D(F(5-))(v,z) = 5»(DF)(5-(v,z)) . 

If, in particular, ip G S, then also (p 5 (v,t) = ip[8(v,z)) is in S. If V = A'(p) and is the 
homogeneity degree of Dj eP, 

£(P„y/) = ^(P^) . 
Hence £x> is invariant under the dilations on IR d 

(2.9) e = (6, • ■ • , 60 ■— ► (<^6, • • • , <^£j = , (5 > 0) . 

On M d we introduce the homogeneous norm, compatible with the dilations D(5), 

d 

(2.10) lieil = El^ • 

5=1 



2.3. Special Hilbert bases. 

We will privilege homogeneous Hilbert bases p which split as p = (p, , pj, p B , Px, i0 ), where, 



keeping in mind (2.1), 



(i) p j0 is a homogeneous Hilbert basis of V($o) , 

(ii) P| is a system of coordinate functions on 3; 

(iii) p D is a homogeneous Hilbert basis of V{x>) K \ 

(iv) p„, o contains polynomials in {V v (t>) <g> P i/ "(3o)) with z/, z/' > 0. 

Denoting by d 3o , dj, d B , <i 0j30 the number of elements in each subfamily, we split IR d as 

]R d = M d M x IR d -' x R do x R d °->o , 

and set 
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Whenever a unified notation for all invariants on 3 is preferable, we set 

Pi = (Pio'Pi) = (pi (*)>•••> a*, (*0) > 6te0 = ■ 

Since \'(q) = Q , («~ 1 V Z ) for every polynomial q on 3, we have 

(2.11) = P,(C) > 

for every bounded spherical function = This gives the following. 

Lemma 2.4. T7ie canonical projection IT /rom lR d onto IR^- 1 restricts to a surjective map 

n| Ec : E c — >• p,(j) , 

and, /or£ 3 G p,(a), 

= Ufa**) p»(Q = &} • 

In particular, the map which assigns to a spherical function </?£ )W ,^ G S(iV, iT) i/ie uaZue 
p,(C) ^ continuous. 

We observe that p 3 (j) is homeomorphic to the orbit space 
2.4. Dominant coordinates in Ex>. 

Lemma 2.5. Lei £ = (£ 3o , £f, £e, £», 3o ) ^ e a point m Ex>. We have the following inequalities: 

(i) if Pj G ^i(o) ® VHto), with i/.yj > 0, then y < C||£j4 MJ"" ; 

(ii) IKII<C||&||. 

In particular, if £ }0 = 0, then also £t>, 30 = 0. 

Proof. Let p(n) = \v\ 2 , where | | denotes the norm induced by a fC-invariant inner product 
on t>. Denote by X v the left-invariant vector field equal to d v at the identity of N. Then 
X'{p) is the sublaplacian L = — ^X 2 ,, where {ej} is an orthonormal basis of D. Then L is 
hypoelliptic and, for every V\, . . . , v m G D and F G S(N), 

\\Xtn ■ ■ -X Vm F\\ 2 < C Vu .„ )Vm \\L~2 F\\ 2 , 
cf. [12]. For zj, . . . , z m G 30 and F G S(N), we also have, by classical Fourier analysis, 

m 

||^ 1 ---a Zm F|| 2 <a ij ... ) , ro ||A 3 2 F|| 2 . 

Therefore, 

||<9 Z1 • • • d Zm ,X vl ■ ■ ■ X Vm F\\l = I (d 2 Zi ■ ■ ■ d 2 ml F, X V1 ■ ■ ■ X Vm X Vm ■ ■ ■ X Vl F) | 

< C Vl> ... )VmtZu _ jXm ,\\L m F\\ 2 \\A™ F\\ 2 ■ 

If Pj G Vi(\>) ® (30), then = \'( Pj ) is a linear combination of terms of this kind, 
with m = v'a and m! = u'L Therefore, for F G S(N), 



(2.12) \\DjF\\ 2 < C||L^F|||||A^F 115 



1 „« 1 

2 
2 
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We may assume that p v contains the squares pk = \vk\ 2 of the norm restricted to mutually 
orthogonal irreducible components of t). Then p is the sum of such p^. The same can be said 
about q(z ) = \z \ 2 on 3 . 

Denote by the component of corresponding to \'(pk), the component of £ i0 cor- 
responding to X'(qi), and ^ the component of £ Bi30 corresponding to Dj. Then, since \'(pk) 
and A'(^) are positive operators, we have £,k,£,e > in H v . Hence (2.12) implies that, on Ex>, 



u<c{Y,^Y '(£&) 2 ^ii&Au 



This proves (i). To prove (ii) it suffices to prove that ||£ 3 || < C||£ D ||. For this, it suffices 
to observe that every derivative <9" in the 3-variables can be expressed as a combination of 
products X Vl ■ ■ ■ X Vm with m = 2\a\. Then, for every F G S(N), 

\\d"F\\ 2 < C\\L lal F\\o . □ 



2.5. Quotients in 3 and Radon transforms of functions and differential operators. 

Given a subspace s of 3, denote by n' the quotient algebra n/s and by N' the corresponding 
quotient group. 

Given a .fT-invariant inner product on 3, set 3' = s -1 and let proj be the orthogonal 
projection of 3 onto 3'. Then n' can be regarded as © 3' with Lie bracket 

[v,w] n > = pro}[v,w] . 

We denote by K' the stabilizer of s in K and fix Lebesgue measures dv, dz', ds on d, 3', s, 
respectively. We define the Radon transforn^TZF of a function F G S(N) K , as the function 
on N' 

(2.13) KF(v,z') = I F(v,z' + s)ds . 

J s 

Then 1ZF G S(N') K \ and for every bounded function G on TV', 

HF{v, z')G(v, z) dv dz = I F(v, z)(G o pro])(v, z) dv dz . 
N' Jn 

Accordingly, given D G B(N) K , we define 1ZD G B(N') K ' as the operator such that 

(2.14) ((1ZD)G) o proj = D(G o proj) . 
Notice, however, that (N', K') is not a Gelfand pair in general. 

We remark here some basic properties of Radon transforms of differential operators, cf. 
[H], Section 4 and Lemma 4.2. Further properties will be stated when needed. 

Proposition 2.6. 



4 The term "Radon transform" is abused here. It comes from the special case where 3 = R", K — SO n and 
= K n_1 . Assuming if-invariance of F, the integrals in (2.13) give the values of integrals over all subspaces 



of 3 obtained from s by translations and rotations by elements of K. However, TZ is not injective in general. 
It is injective if 3' intersects almost all if -orbits in 3. 
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(i) If D = X' N {p) G B(N) K , then HD = \' N ,(p\ nl ). 

(ii) If (N 1 , K') is a G elf and pair and <p' is a bounded spherical function on N' , then 



(2.15) 



Aip'{v,z) — / {ip' o proj)(Aru, kz) dk 



K 



is a bounded spherical function on N . For D e B){N) K , 
(2.16) Z(D,A<f/) = £(KD,(pr) . 



Proof, (i) follows from (2.2). 

For D e B{N) K , set f = £(KD, ip'). Then 



D{Aip'){v, z) = / D{ip' o proj) {kv, kz) dk 



K 



K 



{{lZD)ip') o proj(fct>, kz) dk 
iAip'{v,z) . 



This proves that Kip' is spherical and that (2.16) holds. 



□ 



Since the operation in (2.15) is continuous in the compact-open topology, we have the 
following. 

Corollary 2.7. Assume that {N ; ,K') is a Gelfand pair. The map A is continuous from 
Z{N',K') to E{N,K). 



3. Reductions 



In this section we collect three preliminary results which will be used repeatedly in the 
course of this paper. Moreover, Propositions |3.1 and 3^ are the ingredients that allow to 
reduce the full Vinberg list to Table [1} 

We refer to [9] (which extends an argument of [2j), for the proof of the first result, con- 
cerning normal extensions of K. 

Proposition 3.1. Let K# be a compact group of automorphisms of N, and K a normal 
subgroup of K# . If {N,K) is a n.G.p. satisfying Property (S), then also {N,K#) satisfies 
Property (S). 

The second result, concerning central reductions of N, has been announced in [TU] without 
proof, and we give its proof below. 

Assume that 3 contains a nontrivial proper i^-invariant subspace s. If n' = n/s, as in 



Subsection 2.5, then K' = K. It is proved in [23] that {N',K) is a n.G.p. 



Proposition 3.2. Suppose that Property (S) holds for the n.G.p. {N,K). Then Property 
(S) also holds for {N',K). 
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Proof. We fix a F-invariant complementary subspace 3' of s in 3 and adopt the notation of 
Subsection 12.51 

Given a F-invariant function / on N', the function / = / o proj is F-invariant on N. 
Hence, if (p is spherical on N', the map A in Corollary 2.7 , reduces to composition with proj, 
and therefore it is injective. With suitable choices of generators for B)(N') K and D(iV) , this 
injection can be viewed in the following way. 

We fix a fundamental system (pi, . . . , p$) of ^-invariants on t) ©3', and complete it into a 
fundamental system (pi, . . . , p^, Pd'+i, ■ ■ ■ , Pd) of F-invariants on n, where each of the added 
polynomials pd'+i(v, z' + s), . . . , pd(v, z' + s) vanishes for s = 0. 

Applying the symmetrisation (2.2) on N and N' respectively, we obtain the generating 
systems of differential operators 

V= (D 1 ,...,D d ) , V = (D' 1 ,...,D' d ,) , 

on A" and N' respectively. In particular, 



TZD j 



D) ifj = l,...,d>, 
iij = d'+l,...,d. 



It follows from Proposition 2J3 that, if <p is a bounded spherical function on N' represented 
by the point £' G Ex>/ C M. d , then (p is represented by the point (£', 0) G Ex> C WL d . 

Hence the injective map of Corollary 2.7 corresponds to the map £' 1 — > (£',0) from E75/ 
into E75. The corresponding restriction operator is a continuous linear mapping from <S(Ed) 
onto «S(Exi/). 

Let and be the spherical transforms corresponding to E© and Ex>'. We fix a smooth 
and compactly supported function ip on 5 with integral 1. 

Given F G S(N') K , for any bounded spherical function <p of (A/"', if) we have 

F(t>, z')tp{v, z) dv dz = I F(v, z')ip(s){p{v, z + s) dv dz ds , 

J n 

where (p = p o proj. Thus, for any £' G Exv, 

£'F(FO = £(i^)(£',0) , 
where (F (g> + s) = F(t>, z')ip{s). We have obtained the identity 

(3.1) £'F=(£(F®^)) . 

Obviously, F®ip G S(N) K and the map F 1— >■ F <8> ^ is continuous and linear from S(N') K 
to S(JV)*. 

As we are assuming that (N,K) satisfies Property (S), Q : S(N) K — > <S(Ex>) is an isomor- 
phism of Frechet spaces. By composition, the map 

F^(S(F®V0), = S'F 

is continuous from S(N') K to <S(Ed/). Since we already know that the inverse of Q' maps 
<S(Ex>/) into S(N') K continuously [2], we have the conclusion. □ 



Our third result concerns product pairs. 
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Proposition 3.3. Let (Ni,Kx), (N 2 , K 2 ) be two n.G.p. satisfying Property (S). Then also 
the product pair (N\ x N 2 ,K\ x K 2 ) satisfies Property (S). 



In order to prove Proposition 3.3[ we need the following property of decomposition of 



Schwartz functions on the product of two Euclidean spaces. We use the following Schwartz 
norms on 5(IR n ): 

||/|| 5 (r«),m = II/IIm = sup (i + \x\) M \d a f(x)\ , Mel. 

\a\<A4,x&R n 

Lemma 3.4. Let m and n 2 be two positive integers. Set n = n\ + n 2 . 

Let also ip u , u — 1,2, be smooth function on IR n,y , supported in [—1, l] Uv and satisfying: 

< ip v < 1 and ip u — 1 on [ — -, -] n " . 
For v = 1, 2, l u , m u G Z n " ; set 

Then the following properties hold. 

(a) Let v — 1 or 2. Given M G N there is a constant Cm > such that, 
(3.2) Vl u ,m u G Z n » , \\H^ m Js(^),M < C M (1 + \Q + \m u \f M . 

(b) For any function F G 5(1R' 1 ) there exist coefficients c^ m G C, I — (h,l 2 ), m — 
(mi,m 2 ) G Z n = Z ni x Z™ 2 , snc/i tfm* 



(3-3) F= 



(2) 

LmeZ 11 



(c) For am/ M G N, i/iere is a constant Cm, independent of F, such that the coefficients 
c, m satisfy: 



(3.4) V/,m G Z n |c,, ro | < Cm||F||5(K"),m(1 + |*| + H) 



-M 



Proof. For ^ = 1,2, we consider a partition of 1R™ 1 with the cubes l v + [— |, |] n ", G Z^ 
and the p 
such that 



2' 2J 

and the partition of unity obtained from a smooth function <f> v on M. nv supported in [— |, |] n " 



0<<Pu <1 , 0„ = 1 on [-i i]"" and Vi,el n " ^ ^(^ + Z„) = 1 . 



4' 4 J 



For each I — l 2 ) G Z ni x Z n2 , the function x = (xi, x 2 ) F(x\ — x 2 — l 2 )(f)i(xi)(f) 2 (x 2 ) is 
smooth and supported in [— |, |] n . We keep the same notation for its 27r-periodic extension 
in each variable; then its decomposition in Fourier series gives: 

F(x - /)0 1 (x 1 )0 2 (x 2 ) = d, m e- ix - m , x G i-7c,n} n . 
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It is easy to see that the coefficients q jTO satisfy (3.4) for any M G N. We can write: 

F(x) = F(x)Mxi + h)Mx2 + h)Mxi + h)Mx2 + l2) 

i=(i 1 ,/ 2 )ez n ixz n 2 



E 

:,meZ" 



-i(x+l).m 



satisfy (3.2) and (3.3). 



□ 



Corollary 3.5. For v = 1,2, let K v be a compact subgroup of GL nv (M). For v = 1,2, 



l u ,m u G Z™", t/iere exist K u -invariant smooth functions H i 
elusions of Lemma 3.4 hold, with H l 



l v ,m v 



m 



place of H^ mv , for F in S(R n ) and K x x K 2 - 



on 



such that the con- 



invariant. 



Proof. Just take as H^ mi/ the ^-average of H\"J mv - The conclusion is quite obvious. □ 

Proo/ o/ Proposition [Oj We fix two families, Z?W = (L>f } , . . . , D^) and £>( 2 ) = (£>f \ . . . , D^), 
of generators of ©(iVi)^ 1 and 3(N 2 ) K2 respectively. We keep the same notation when the 
operators are applied to functions on N = N x x N 2 by differentiating in the iV^-variables. 



ThenP = (D^ , . . . , D ( d \> , Df 1 , . . . , D%) is a family of generators of B{N) K , K = K x x K 2 



r>( 2 ) 



)(2) 



X Et>, C 



adi x R d 2 



— ^V 2 

and, if Qi, Q 2 , Q are the corresponding Gelfand transforms, then 

Q{F\ ® f 2 ) = {g 1 F 1 ) ® (g 2 F 2 ) . 

Identifying N% and N 2 with their Lie algebras, we consider th e -^ -invariant functions 
ff, L , u = 1,2, l v ,m v G Z, n ", satisfying the properties of Corollary 



lv,m u ' 

Given F G S(N) K , we decompose it as 



3.5 



(2) 



m 2 



with coefficients Q :Tn satisfying (|3.4|). Then 



m 2 ' 

Since we are assuming that each (N V ,K V ) satisfies Property (S), given any MeN, there 

for v = 1,2 and Z^, G Z n ", and an integer Am such that 



are functions h,^ G 5 



W coincides with Q v H\J mv on S p „ 



fii) life 



(i/,M), 



i, m ,; ii5(K^),Af < CmII^^IIs^),^ < Cm(1 + %\ + \m v \) M . 

If we set 

AM) _ ,(1,M) ,(2,M) 
n l,m — rl h,m 1 n l 2 ,m 2 

where d = di + d 2 , we have, for any I, m G Z n , 



G §(M d ) 



(3.5) 



%?l|5(K<'),Af < Cm(1 + |i| + H) 



2A» 
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Combining the rapid decay of the coefficients with the polynomial growth (3.5) of the 
-th 

(3-6) 



M-th Schwartz norm of the , we obtain that 



\ c l,m\\\hi^\\s(EL*),M — C-M\\F\\ S ^ N ) t B A 

with B M = 2A M + n + 1. 

Given Mo G N, we want to construct a Schwartz extension /( M °) of C/.F whose Mo-th 
Schwartz norm is controlled by a constant, independent of F, times 11^115(^,5^0 ■ 
By (3.6), for every M > M , there exists au = o>m,m ,f G N such that 

-m\\ f \ 



(3.7) 



|5(JV),B Mo 



,m e Z" 
- |m| > om 



The czm can be inductively chosen to be non-decreasing. Then we define as 

oo 



+ M < a Mo+1 



J2 Yl ci > mh Z 

m=m +i i )m ai n 

a M < \l\ + M < a M+1 

Clearly, the series converges on to QF. To show that it defines a Schwartz function 
on all of lR d , notice that, for every Mi G N with Mi > M and every M > Mi, we have by 
gjb that 



5^ Icj.fulll^ 1 



^ i iiia( m )ii 

m ' ||5(Kd),Mi S 2^ R,™lll ft Z,m ll5(IR d ),Af 



< 2- M ||F| 



S(N),B Mq ■ 



a M < \l\ + \m\ < a M+ i 

This implies that 
have that 



as required. 



i,m e z n 
OM < I'l + M 



5(R d ),M : 



•j is finite. Moreover, combining (3.6) and (3.7) together, we 



11/ 



(Af) I 



S(R d ),Mo 



< (C M() + 2 



|5(JV),B Mo 



□ 



4. Pairs satisfying Vinberg's condition 



Given the results of [H El El E] and the reduction arguments exhibited in Section [3j the 
pairs for which we need to prove Property (S) are those listed in Table [I] We recall the 
splitting 3 = 3o © 3, where 3 is the subspace of fixed points under K. 

All pairs in Table [T] admit Hilbert bases p which are free of relations and satisfy the 
conditions of Subsection |2.3[ They are given in [91 Section 7] and are reproduced in Table [3] 
at the end of the paper. For unexplained notation, we refer to [§]■ 

It is apparent in Table [I] that K = K x K' (with K' possibly trivial), where the action 
of Kq on 3 is faithful (up to a finite subgroup at most) and that of K' is trivial. 

We say that an element of 3 is regular if its il~-orbit has maximal dimension and singular 
otherwise. Clearly, regularity of an element only depends on its 30-component. 
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K 





3 


notes 


3o (if t 3) 


1 


so n 


TTT) T? 

K 


so n 


n > 4 




o 
Z 


TT 


(Pn 




n ^ z 




3 




e n 


ijs 2 e n ©ime 


n>2 


HS%U n 


4 


SU 2n+ i 


£<2n+l 


A 2 C 2n+l 


n>2 




5 


U2n+1 


(J~«2n+1 


A 2 C 2n+l R 


n > 1 


A 2 C 2„+1 


6 


su 2 „ 


C 2n 


A 2 C 2n © R 


n > 2 


A 2 C 2n 


7 


U 2 x SU n 


c 2 ®c n 


u 2 


n > 2 


5U 2 


8 


U 2 x Sp n 


C 2 <8) C 2n 


u 2 


n > 2 


SU 2 


9 


Ui x Spin 7 


IR 2 <g) O 


ImO©i 




ImO 


10 


Sp 2 x Sp n 


H 2 ® H n 


Sp2 


n > 1 





Table 1. The reduced list of pairs satisfying Vinberg's condition 



5. Quotient pairs 



Given t G 3, we set i t — t-t, i.e., the tangent space in t to the K-orbit K-t, and 3t = (t -t)" 1 . 
We consider the quotient algebra n t = n/t t , and denote the canonical projection by proj t . 



As in Subsection 2.5, we regard n t as D ©3*, with Lie bracket [f,"i/] ni = P r ojJf,f']. By N t 
we denote the quotient group N/ expt t . 

Since tf and it are invariant under the action of K t , the stabilizer of t in K, passing to the 
quotient we obtain an action of K t on N t . We call (N t ,K t ) a quotient pair of (N,K). By 
(A^t) is an.G.p. 

The quotient pairs generated by the pairs in Table [T] are listed in the Appendix at the end 
of the paper. Here we only point out the following facts. 

Remark 5.1. For the pairs at lines 1-6, the quotient pairs are products of pairs in the same 
list, possibly up to a central reduction. In detail, pairs at line 1 bring in pairs at line 2 plus 
pairs at line 1 of lower rank. Pairs at lines 2 and 3 form self-contained families. Pairs at 
lines 4, 5, and 6 lead to pairs at line 3 and pairs of lower rank in the same family. 

For the pairs at lines 7-10, we obtain instead new pairs, which are indecomposable, but 
do not satisfy Vinberg's condition. 



5.1. Slices and radialisation. 

Suppose that a compact real Lie group K acts on a linear space W. Take any x G W and 
let K x be the stabiliser of x in K. Let also N x := (fi • x) 1 - be the normal space to the orbit 
Kx at x. A construction of a slice for a compact group action goes back to Gleason [13]. 
We will need the "linear" version of the slice theorem. 

Theorem 5.2. There is an open and K x -invariant (Euclidean) neighbourhood S x ofO in N x 
such that the K-equivariant map 

a:Kx Kx S x -^W, 
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given by a(k,y) = k(x + y), is a diffeomorphism of K x^ x S x onto the open neighbourhood 
K(x + S x ) ofKx. 

We call S x a slice at x. The notation K x Kx S x stands for the quotient of K x S x modulo 
the action of K x , i.e., (kk',x) is equivalent to (k,k'x) for k! G K x . 



The proof of Theorem 5.2 can be found, for instance, in [H Ch. 2, Section 5], in particular, 
see Corollary 5.2 therein. The theorem has the following almost immediate consequence (for 
part (i), see e.g. [U Ch. 2, Sections 4, 5]). 

Corollary 5.3. 

(i) For every y G S x we have the inclusion K y C K x , more explicitly K y = (K x ) y . 

(ii) Two points in S x are conjugate under K if and only if they are conjugate under K x . 

(iii) Suppose that f is a K x -invariant smooth function on S x . Then f extends in a unique 
way to a smooth K -invariant function / rad on K(x + S x ). 

We call / rad the radialisation of f. 

Remark 5.4. This notion of radialisation extends the one used in [8] to general pairs, 
including the rank-one pairs considered in [9J. We notice that the arguments in the rest of 
this paper will not rely on the results of j9], which in fact are being given a different proof. 



We will apply Theorem 5.2 and Corollary 5.3 to the action of K on 3. If t G 3 and St is 



a slice at t in 3, then D x St is a slice at t for the ^-action on ^©3. (The injectivity on the 
"3"-side and zero at the 'V-side of t allow us to include D into the slice.) 



5.2. Relations between invariants for (N, K) and invariants on its quotient pairs. 

For t G 3, we consider the quotient pair (n 4 , K t ) defined at the beginning of this section. 

Let p, resp. p*, be a minimal Hilbert basis for (n,K), resp. (n t ,K t ), whose elements are 
homogeneous in both the D- and the 3-variables. Denote by p^, resp. p* fc ^ the invariants in 
p, resp. p*, which have degree k in the D-variables. Then p^ = p, and p' ^ = p* t . 

Remark 5.5. We point out some identities concerning cardinalities of Hilbert bases for a 
given pair and its quotient pairs. Strictly speaking, none of these facts is needed in the 
sequel. We will nevertheless assume them, both for notational convenience, and to avoid 
unnecessary inductions on k in some of the proofs in this section. 

(i) The cardinality of each of the sets p^), p^ does not depend on the choice of the 
minimal Hilbert basis; 

(ii) for every k, p^ and p^ have the same number of elements; 

(iii) the different values of k that appear are at most three, either 0, 2 or 0, 1, 2 or 0, 2, 4. 

To have a unified notation, we will denote the values of k in (iii) as 0, ko and, when 
present, 2k . 

The first statement is perfectly clear, for example, the cardinality of pi k \ is equal to the 
dimension (over K) of A k /(A k C A 2 ), where A = (& >0 (n)) K and A k = An 6 fc (o)<g>6(3). 
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Since the inaction on n admits a free Hilbert basis, see [91 Section 7], this property is 
inherited by (nt,K t ), see e.g. [TBI Theorem 8.2]. In particular, a minimal Hilbert basis p* is 
also free of relations. Recall next that i^-orbits in n (as well as ^-orbits in tit) of maximal 
dimension form a Zariski open subset, the so-called subset of generic points. Being a non- 
empty open subset, t>xS t contains a generic point, say (, of n t . By Theorem |5.2 ( is also 
generic in K(\jx(t + S t )) and hence in n. We see also that ^ = (j t )^ and Kq = (Kt)t- Since 
orbits of a compact group are closed and closed orbits are always separated by polynomial 
invariants, the number of elements in p^ (resp. p\o)) is equal to the codimension of a 
generic K-orbit in 3 (resp. fQ-orbit in $ t ), i.e., diny^ (dim (3^, respectively), see e.g. [JSJ 
Sections 2.3, 4]. This takes care of the case k = 0. For k > 0, the statement of part (ii) 



follows now from Lemma 5.6 applied to both (K,n) and (K t ,n t ). 
Statement (iii) is evident from Table [3l and Table |2l in Section |9 



Lemma 5.6. Suppose that we have a linear action of a compact group K on D©3 and the 
algebra of invariants W\p@^\ K has a free Hilbert basis. Then the number of generating bi- 
homogeneous invariants of degree k > in D is the same as the number of generators of 
degree k in a minimal Hilbert basis in m ' r "' A 
maximal possible dimension. 



0] c , where £ G 3 and the orbit C 3 has the 



Proof. In view of Theorem 5.2, the maximality of dimK( implies that the action of on 



3^ is trivial. Consider now the composition 



\n] K -> 



D©a c ]*< ->• R[0 x {(}] 



of two restriction morphisms. It is known to be surjective, see e.g. [281 proof of Theorem 1.3]. 
The elements of E[j] are constant on x {£}. Since i^-orbits are closed, tr.degM^]^ = 
dim3 — dimK( (the codimension of a generic orbit, see e.g. [IB1 Sections 2.3, 4]) and 
since the whole algebra of i^-invariants on n is free, there are exactly dim 3^ generators 
in Kfo]^. The remaining generators of R[n] restrict to the generators of Mft)]^, because 
tr.deg]R[n] A — tr.degIR[t)] Xc = diny^ by the same minimal codimension reasoning. □ 



' Pk,d k 



We label the elements of p^ as (pk,i, ■ ■ ■ , Pk,d k ) and those of p*^ as (pj.,n ,,, 
When we restrict an element of p to tit, it can be expressed as a polynomial in the elements 
of p*. By degree considerations, it is quite clear that 



(5.1) p kJ] (v,z) 



'<2o,j(p( )W) 



lfe 



[v,z 



d-2k 



J2Q2k ,j,t(p\o)(z))p t 2koA V > Z ) 



d kQ 



if k = 
if k = k 

if k = 2k n 



+ X RvotM Wo)( z ))pI>. z )PhoA v > z ) 



ii'=i 
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where the Q's and the i?'s are polynomials. Then the map Q from M. d = M. do+dk o +d2k o to 
itself, whose components Qkj are 

Qo,j = Qoj(£(o)) ) 

(5.2) 

<^2fc d k() 

Q2k ,j = ^ Q2fco,j^(^(0))^2fc / + ^ R2k ,j/,i'{i(0))ik ,iik 0) & , 
£=1 £,£'=1 

maps p*(o x (t + S't)) into p(o x + S^)). Since the elements of p*(o x [t + S t )), resp. 
p(t) x iC(£ + St) J , are in 1-to-l correspondence with l^Q-orbits in D x (t + S t ), resp. i^-orbits 
in D x K(t + 5* 4 ), it follows from Corollary 5.3 that Q is a bijection from p*(t) x (t + St) J 
onto p(t) x + St)). In particular, <2 

p 3 (AT(t + $))- 

We show now that Qr x is the restriction of a smooth map on a neighbourhood in W 1 of 
p (0 )(t) x M dfc o+ d 2fc . 

Proposition 5.7. Fort e 3, Ze£ fre as in Corollary \5. 3 , and let U be a Euclidean neigh- 
bourhood of t, K t -invariant and relatively compact in t + S t . Then there exists a smooth 
function <£> from R d to R d which inverts Q on p(n) fl (pr \(U) x M^o+^o) . The scalar 
components §k,j of $ have the following form: 

(i) each $ ,i on ^U depends on £( )/ 

(ii) $fc ,;(0 = E£°i $fco,i,^(o))6w; 

(iii) $2fe ,i(0 = ^£=1 $2fc j\£(£(0))6fc ,£ + Z)^'=l *2fo,j,£,£'(£(0))6fc ,£6fco,£'- 

In particular, $ = (^oj)i<j<d 3 inverts <2 on pJU). 
Proof. Let % G C^°(t + S t ) be ^-invariant and equal to 1 on U . 



(Qo,j)i<j<d m is a bijection from p\ t {t + 5* t ) onto 



u j( z ) = (xPj) ( z ) on K(t + St) and extend it to 3 as 0. By Corollary 



? or j = 1, . . . ,d 3 , set 
5.3 (iii), Uj is smooth, 

so that, G. Schwarz's theorem [19], Uj = $ j(pi, ■ ■ ■ , pd s ), with $ ,j smooth on M d ° = I^ 3 . 
Then 

(5.3) p) = $ j(Pi,- • • ,Pd s ) , j = l,...,d,, 

on [/. Combining this with the first line of ( 5.1| , we conclude that the map 

< ^o = ($o J )i< J <d, :p 3 (^)^p; t (^) 
is the inverse of the map Qq. 

We apply now the same construction to the elements of pa. y The radialisation of xp\ p 
defined on D x 3, remains a polynomial of degree ko in the D-variable. Applying Schwarz's 
theorem again and using Proposition 2.1 of [ID] (with the roles of t) and 3 interchanged), we 
have that 



(Xpij^ = J2^ k oAf P(0))P 



ko,j' ' 
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with $ ko ,i,j> e C°°(R d »), and this gives (ii). 

In the same way we obtain, for a polynomial in P( 2 fc ), that 



d (2fe ) 



(XP2fe ,£) rad = ^ ($2fcoAf' ° P(0))P2fe ,j' 



i'=i 



+ E ^k ,i,j',j" ° P(0))Pk ,j'Pk ,j" , 

f,j"=l 

with $2kQ,e,j',^ko,t,j',j" € C°°(lR d3 ), and this gives (iii). □ 



6. Smooth maps between spectra 



Let (AT, A') be a pair in Table [T] Following the notation of (2.15), to a bounded spherical 
function for the pair (A^, A^), we associate the bounded spherical function for (N, K) 



(6.1) 



A t ip t (v,z)= / (ip* opioj t )(kv,kz) dk . 



K 



Proposition 6.1. Then A* is a continuous surjection o/E* onto E. 

Proof. Given £ e a and w G t( , consider the two decompositions of ?^ into irreducible 
components under the action of K^ jU and (A^)^ respectively, 



v) 



vex 



Since both decompositions are multiplicity-free and (Kf)^ C AT^, each l^(yu) is a finite 
union of V t (u). 



The spherical function <£>£ )W)I , £ E* is, by (2.5), 

1 



tr (7T C)U (At;,^)[ v ) dh . 



dimV^(z/) 

By Schur's lemma, for every \i G and every unit element e € ^(/-t), 

1 



{^c„^{kv, kz)e, e) dk 



K 



tr (>K CtU1 (kv,kz) lvw ) dk . 



dimV(/j) J K 

It follows easily that, if V t {y) C then A*</?£ w = ^C^m- Continuity of A* with 

respect to the compact-open topologies on the two spectra is obvious from (6.1). □ 

We point out that the proof shows the following identity: 

(6-2) (Ar^VW) = H.,u ■■ V*(u) C . 

This has the following consequence. 
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Corollary 6.2. For t G i, let St be as in Corollary 5.3\ If 
(6.3) St = {^:Cet + St}cS , 

then A* is a bisection from (A*) _1 (S't) to St- 



Proof. For ( e S t , K c = (K t ) c by Corollary 5.3 (i). Hence, by (6.2) and (2.5), A* is 



injective. 

The same proof shows that the following more general statement holds. 
Corollary 6.3. Ift,t' e 3 and K t C Kf (hence it C it' ), the map 

(6.4) Mj f <p t (v,z)= ((p t o W o2 t )(kv,kz)dk , e 3*0 

is continuous and surjective from E* to E* , ond 

(6-5) (A^r = : f» ^ • 

Ift,t',t" are such that K t C C i£t» i/ien 

A^„ = Aji o A*, . 



□ 



Moreover, S t satisfies the conditions of Corollary 5. 3 for the action of K t > on if and A^, 
zs invertible on (A*) _1 ( ( 9 t ) ; mt/i i? t £/ie set m (6.3). 



The whole picture is represented by the following "dual" commutative diagrams, repre- 
senting the different "levels of singularity" of elements of a. We then refer to the elements 
of 3 as the "most singular" elements of a. The sub- or super-script "reg" stands for regular 
elements. 



proj r 



it 



D © a 



P ro Jreg 



E* 



Vjreg 



V7 

E*' 



D © It" 
P ro Jt" \ 



© 3t"' 

/ proj t /// 



E*" 

A'" \ 



E* 



E* 



Assuming that two homogeneous bases T>, resp. V t , of ©(AQ^, resp D(iVi) *, have been 
fixed, we regard the map A* in ( |6.1[ ) as a map from E^ to Ex>. 

We want to realize A*, and its local inverse on St, as restrictions of smooth maps on 
open subsets of the ambient spaca^j Notice that the existence of such smooth extensions is 
independent of the choice of V and V t , because of (2.7). 



Denote by 1Z 1 the Radon transform (2.14) adapted to the projection proj t of n onto tit. 



By (2.16), the eigenvalue ^(TZ t D,ip t ) is the same as the eigenvalue of £(-D,A*<^ 



'We do not require these extensions to be diffeomorphisms of such open sets. 
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Proposition 6.4. LetT>, resp. T> t , be homogeneous bases ofB>(N) K , resp. B>(N t ) Kt , consist- 
ing of d, resp. d! , elements. Then A*, regarded as a map from HL to Ex>, is the restriction 
of a smooth map from M. d to M. d ' . 

Proof. Let D = (Z) 1; . . . , D d ). Since the statement does not depend on the choice of V t , we 
construct one starting from (11* Di, . . . ,TZ^D d ) and completing it into a generating system 

v t = (n t D 1 ,...,n t D d ,D t d+1 ,...,D d ,) 

of 3(N t ) Kt . By Proposition 2.6, A* is the restriction to Ex> t of the canonical projection from 



l d ' to its first d-dimensional coordinate subspace. □ 
From now on, we denote by T>, resp. V t , the system of differential operators obtained by 



symmetrisation, resp. on A" and N t , from the Hilbert bases p, resp. p , of Section 5.2 Then 



S c and E^, t are subsets of the same space M. d . 

for each point £ 3 G p 3 (3), there is an associated subset of Ex>, denoted 



2.4 



By Lemma 

by LTr 1 (£), whose elements £ D , £0,30) correspond to the spherical y^C.^.M in (2.5) with 



P 3 (C) — Cj- The same applies to E|, t and a point £ jt G p* t (3 t ). 

Proposition 6.5. Let t G 3, and let U C St be as in Proposition 5.7, Then A* coincides with 
a map Q, obtained from Q in (5.2) by adding a linear term in C,^ ) to the components Q2k ,j- 
Then Q is still invertible on Ex> t fl IT -1 (p* ([/)), and (A*) -1 = Q~ l is the restriction of a 
smooth map $ 7 obtained by adding a linear term in to the map $ of Proposition 



5.7 



Proof. Take a point £ G Ex> t with . . . , £ d J G p, t (U), and let £ = A'£ G Ex>. 

If <p* is the AVspherical function on N t associated with £, then <p = A*<p*, is the A'-spherical 
function on A" ass ocia ted with £. 

^£if for Di G X>, we also have that (TV'D^ip 1 



By Proposition 
Moreover, 



2.6 



since -D^tp 



^ J D, = A / (p, k ) = A / (g,(p t 1) ...,p*)) 
where Qe is the polynomial in ( |5.1 ). 



Notice that, by Q, A'(Q,(pi, . . . ,p*)) differs from Q e (D{, . . .,D% i.e., Qe(\'(p\), . . 
only for the components of degree 2/co, in the terms containing a product of two Pk ,j- 
sufficient for us to observe that 

The conclusion follows easily. 



+ A'(linear combination of elements of p^ )) ■ 



to- 



,A'(p 
It is 



□ 



7. Extending Gelfand transforms: reduction to quotient pairs 



Denote by So(N) the space of function F G S(N) with vanishing moments of any order 
in the 3 - variables, i.e., such that 

z@F(v, z, u) dz = , 



jo 
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for every /3 G N ds °, v G D, u G 3. 

In this section we prove the following statement. 

Proposition 7.1. Le£ (N,K) be one of the pairs in TableUi and assume that Property (S) 
holds for all its proper quotient pairs (N t , K t ). If F G So(N) , then its spherical transform, 
defined on S©, can be extended to a function in <S(R d ) which vanishes with all its derivatives 
on {0} x R d > x ]R d » x R do >*o. 

The set {0} x M.^ x ]R d ° x R do '*° coincides with n _1 (p 3 (3)). Its intersection with £x> is 
the set S^, of points with "highest level of singularity", i.e., associated to trivial orbits in 
3, cf. [TO]. Under the identification of £ with Ex>, it corresponds to the set of spherical 
functions which are identically equal to 1 on 30, or, equivalently, which are associated to 
representations of N that are trivial on exp3o- 



To prove Proposition 7.1 we will work on the complement of Sfp, i.e. at points which are 



"regular" or of "intermediate singularity" . It is on neighbourhoods of these points that we 



have at our disposal the local identification of Proposition 6.5 with spectra of quotient pairs. 



7.1. Dilations and partitions of unity. 



Coherently with the previous comments, in this section we restrict our attention to ele- 
ments t of 3 which are not in 3. Then (N t ,K t ) will be a proper quotient pair of (N,K). 



The constructions in Sections |5.2| and [6] present a natural homogeneity with respect to 
the dilations on 30, as well as a translation- invariance in 3. Precisely, if U C t + St is the 
neighbourhood of t 7^ in 3 t for which Propositions 5.7 and 6.5 hold, then the same hold for 

(i) the neighbourhood 5U of 5t, for 5 > 0; 

(ii) the neighbourhood U + u of t + u, for u G 3. 

Also notice that, since K acts trivially on 3, the 3- variables only appear in the components 
of p- } and p h . 

All this has the following implications on the maps of Propositions 5/7 and 6J3, which we 
denote now by Q t , § t , Q t , & t . 

(i) The maps Q t , $t ; Qt, &t contain the identity function in the ^-components, and all 
the other components do not involve the ^-va riables. 

(ii) In the statements of Propositions 5.7 and 6.5 we may assume that t G 30 and choose 
S t and U of the form S t 



So,t + 3> with S 0>t = S t H 30, and U 
neighbourhood Uqj of t relatively compact in t + Sq^. 



U 0>t + 3, for a 



(iii) Let D(5), resp. D t (5), be the dilations (2.9) on R , with exponents Uj, resp. z/*, 
equal to the degrees of homogeneity of the elements of p, resp. p l . Then, for 5 > 0, 
Qst = Qt, Qst = Qt and 

D(5) oQ t = Q t o D\5) , D{5) o Q t = Q t o D\5) , 



(iv) once $t, $t have been chosen for t with \t\ 



(7.1) 



$ 5t = D\S) o $ t o D(5~ 



/>7 



1, $,54, &st can be chosen, for 5 > 0, as 
= D\5) o$ t oDfr 1 ) . 
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Let T be a finite set of points on the unit sphere in 30 such that {KUo y t}teT covers the 
unit sphere. Then there is r > 1 such that the annulus {zq G 30 : 1 < \zo\ < r } is contained 
in [J t T KU 0)t . Therefore {r 3 * KUo )t }teT ,j& is a locally finite covering of 30 \ {0}- 

For each t G T we choose Xt > in C^°(?7o,t) so that Xt > on {z G 30 : 1 < \z\ < r}, 
and set 

Up to dividing each xTj by zJteT jez Xtj> we ma y assume that the xTj form a partition of 
unity on 30 \ {0} subordinated to the covering {r J KUo t t}teT,jez- 

Lemma 7.2. There exists a family {rj t j}teT , jez of nonnegative functions on Mr*o and a 
D (5) -invariant neighbourhood Q of P }0 {do) \ {0} in M> ds o such that 

(7.2) ^(0 = 1 > K e n) , 

and, /or every £, j ; 

(i) ^eCr(R^\{0}); 

(ii) rHj(0=Vtfi(D(r-%; 

(iii) (supp^j) n p 3o (3o) C p io {rWo, t ); 

( iv ) Vt,j(pi (z)) = X*(z). 

Proof. There exist smooth functions Ut, t G T, on R d >o such that 

u t (p i0 {z))=x?\z) = xUz) ■ 

Setting u t j = u t o D(r~^), we have xfj( z ) — u t,j {p i0 ( z )) > fo r every t and j. 

Since p 30 (suppx*o) does n °t contain the origin, we may assume that each u t , t G T, is 
supported on a fixed compact set E of M d3 o not containing the origin. Moreover, since p l0 is 
a proper map, and P 3o (3o \ KUoj) is closed in M. ds o and disjoint from p 3o (supp xto)> we ma y 
also assume that (suppw t ) C p 3() (3o) C p }0 (KU ,t) = p i0 {U 0)t ). 

Clearly, Y^tj Ut d = 1 on Pj (3o) \ {0}- Therefore, if we set 



Vt,j — u t,j on p 3o (3o), and the sum of the r] t j remains equal to 1 where some r\ t j is positive. 
Then (7.2) and properties (i)-(iv) follow easily. □ 



7.2. Characterisations of functions in S (N) K . 

If g is an integrable function on 30 and F and integrable function on N, we set 

(7.3) F* 3o g(v,z, u) = / F(v,z - z',u)g(z') dz' . 



jo 
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This can be regarded as the convolution on N of F and the finite measure Sq © g, where 
Sq is the Dirac delta at the origin in D ©J. We use the symbol ~ to denote Fourier transform 
in the 3o _var i a bles. For a function on N we then set 

(7.4) F(v,C,u)= [ F(v,z,u)e~ iM dz , 



jo 



for v E 0, C e 3o, w e I- For F and # as in fl7^3[ ), F * 3o C, tt) = C, u)g((). 
Finally, we denote by ip t ,j the inverse Fourier transform of xfj- 

Lemma 7.3. The following are equivalent for a function F E S(N): 

(i) F E S (N); 

(ii) F(v, C, u) vanishes with all its derivatives for ( = 0; 

(iii) for every k E N, F(t>, z) = ^| Q | =fc d^G a (v, z), with G a E S(N) for every a; 

(iv) the series J2t&T jez F *io i>t,3 converges to F in every Schwartz norm; 

(v) for every Schwartz norm \\ \\s(n),m and every q E N, \\F * }0 ipt,j\\s(N),M = o(r~ q ^) as 
j — > ±oo. 

Proof. The equivalence of (i) and (ii) is a direct consequence of the definition of Sq(N). The 
equivalence of (ii) and (iii) follows from Hadamard's lemma, cf. [10]. Finally, the equivalence 
among (ii), (iv) and (v) can be easily seen on the 3o _ F° ur i er transform side. □ 



7.3. Radon transforms of F-invariant functions. 



Given t E 3o and F E S(N) K , let H l F E S(N t ) Kt be its Radon transform fl2l3| defined 
on N t . By dO, for ip* E £*, we have, cf. 0, 



7?.*F(f , z')y>*(t>, z') dv dz' = / F(v,z)A t (p\v,z)dvdz 



N t 



N 



i.e., 
(7.5) 



gt{n l F) = (gF)oA t 



We are now able to prove Proposition |7.1 



Proof of Proposition 1_A_ Decompose F according to Lemma 7.3| (iv). Then 

QF= 

teT,jez 

Denoting by fi t j the measure Sq © ip t ,j, where Sq is the Dirac delta at the origin in D © |, 



Q(F * i0 ip t ,j) is the product of QF and GfH,j- By (|2.5), if C = Co + C £ 3o © 3, then 



G to Ave 



in 



A,j( z ) i Pc^A '~ z ^) dz 

^t d {z)e- l{Co ' kz) dk dz 
X*(Co) • 




'30 JK 
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Then, for £ = &,6y ) G E-p, 

(7-6) ^(^* 3o ^i)(0 = ^(eKi(4o) • 

Set = F * }0 iptj, and consider its Radon transform *R}F t j. Since we are assuming that 
Property (S) holds for (N t , K t ), we have GtO&Ftj) G <S(E|, ). Moreover, for every M, q G N, 



jJI|M,5(S*, t ) 



For fixed M, there exist functions h[ h p G «S( 



or 



such that 



h 



(AO 



Let $ r jt be the map in (7.1). Since GF t j is supported in fl LT 1 (p, o (r : 'C/o,t)), (7.5) and 
Proposition 6.5 imply that the composition = o <f>, rJt coincides with on E©. 
Therefore, for every choice of the integers Mj , we can say that the series 

E wAU9% j) (0 

converges pointwise to GF on E©. In fact, many of the terms will vanish identically on E©, 
and this surely occurs when p 3o (r J C/ ,i') H p }0 (r j Uo, t ) = 0. Therefore, if i?tj = {(£',/) : 
P 2 „K^)np 3o (r^ Oit )^0}, 



(7.7) 



E E ^(uC } (o = wo 



on E c . Notice that, by construction, there is A > such that \j — j'\ < A for (t', f) G E t j. 
In particular, the sets E t j are finite and their cardinalities have a uniform upper bound. 

We claim that, choosing the Mj appropriately, we can make the series (7.7) converge to 
the required Schwartz extension of GF. 

In order to estimate the Schwartz norms of T]t'j'9tj j f° r (t'if) e ^tj> observe that, 
by Proposition 5.7, the 3o- var iables are bounded to a compact set, and the other variables 
appear in $ t as linear or quadratic factors. Taking this into account, together with the 
scaling properties of <& rH and cf. (7.1) and Lemma 7.2 (ii), it is not hard to see that 
there is Pm depending only on M such that, for (t', f) G E t j, 

\\Vt',j'9t,j \\M,S{R d ) S or \\ n t,j \\pM,S 



Hence, 



\Vt',j'9t,j \\m,s 



( r -?b1) } 



for every q and (£', f) G E tt j. 

By induction, we can then select a strictly increasing sequence {jg} g6 N of integers, such 
that jo = and 

\\Vt>,f9^\\ q ,s^ <r- q]j] 
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for j > j q and (1?,f) £ E t j. For j q < j < j q +i, we select Mj = p q and consider the following 
special case of ( 7.7[ ): 



9=0 jq<j<jq + l (t',j')eE t J 

teT 



Then the series converges in every Schwartz norm. Since each term vanishes identically 
on a neighbourhood of {0} x M> d ' 3 x IR d ° x l^ '^ the sum must have all derivatives vanishing 
on this set. □ 



8. The pair (N, K) and Taylor developments 



In this section, we complete the proof of Theorem 1.2, under the assumption that Prop- 
erty (S) holds for all proper quotie nt pa irs of (N, K). In order to do so, we want to remove 
the restriction on F in Proposition 7.1 and assume F £ S(N) K . This means analysing QF 
in a neighbourhood of t v = n ({0} x R d i x R d " x IR d »-o). 

8.1. The group N. 



By Lemma 2.4, the spherical functions <fc,<^,^ associated to points in this set are the 
ones for which £ £ 3. This means that the representation 7r^ w factors to a representation 
of N = iV/exp3o and, correspondingly, <£>f, W)j u is a spherical function of the pair (N,K) 
composed with the canonical projection. 

Notice that, as we have implicitly admitted a few lines above, (N,K) is a n.G.p., and 
that Property (S) is known to hold for it (in fact, N is either a Heisenberg group, or its 
quaternionic analogue, with n = HP ©ImH). 

We denote by V i0 , V v , £> B)3o the families of operators corresponding to the correspond- 
ing subfamilies of polynomials in p. 

The following properties are easily verified: 

(i) for D £ "Cjo.f! we have -D^c.^.m = 0' i- e -> whenever £ £ £x> has £ 3o = 0, then also 



£d,3o = 0' c f- Lemma 2.5 



ii) if 1Z denotes the Radon transform (2.14) mapping differential operators on iV into 



differential operators on N, then 1ZD = for D £ T> i0 U T> Vi0 , and P = {TZD : D £ 
T>i U P D }, is a free homogeneous basis of ©(iV)^; 
(iii) if Y,f) is the Gelfand spectrum of (N, K), then 

^•1) ^ = {(0,e i ,£ o ,0):(e l ,e o )£S^}; 



(iv) denoting by ^ the Gelfand transform of (N, K) and by TZ the Radon transform (2.13) 
with the integral taken over 3o ; we have the identity 

^•2) g(nF)(^^) = gF(o,^,^,o) . 
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8.2. Taylor developments on £x>. 

For a multi-index a = (a', a") G N di ° x N^o, we denote by [a] the degree of p" 'Po' 3 ' in 
the 3o- v ariables. 

For pj G p }0 U p VM , let Dj G D(iV) <2)P(3o) denote (A^ ®/)(pj), where A'^ is the symmetri- 
sation operator (2.2), and p.,- is regarded as an element of P(n) <S> P(3o)- If at = (a', a") G 
N d « x N d "- 3 o, then 

D Q = nf f) a " 

has degree [a] in the 3cr var i arj les. 

In [TU] the following result was proved^ 

Proposition 8.1. Let G be a K -invariant function on N x $ of the form 

(8.3) G(v,z,u) = z J G-,(v,u) , 

h\=k 

with G 1 G S(N). Then there exist functions H a G S(N) K , for [a] = k, such that 

(8.4) G = Y] -.D a H a . 

r n a - 

[a]=k 

We complete, giving all details, the argument sketched in [10J, which derives the following 



Hadamard-type formula from Proposition 8A_ 
Proposition 8.2. Let F G S(N) K , and assume that 

F(v,z,u) = d2R~,(v,z,u) , 
\y\=k 

with i? 7 G S(N) for every 7. Then, for every a with [a] = k, there exists a function 
F a eS(N) K such that 

with h a G S(R d i +d °), and 

F(v,z,u) = ^D a F a (v,z,u)+ dPR' Y (v,z,u) , 

[a]=k ' |7'[=Jfc+l 

with R ' , G S{N) for every 7'. 

Proof. Consider the Fourier transform ( |7.4[ ) of F in the 3o-va r iables. Then 

F{vX,u)=i k Y,CR,{vX,u) . 

l7l=* 

Setting G 7 (v,u) = i fc i? 7 (t> , 0, u) G S(N), we have 

h\=k |7'l=fc+l 



6 In fact, [TU] also gives the control of the Schwartz norms of H a in terms of Schwartz norms of the G 7 , 
but this is not strictly needed here. 
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Since G = X^C 7 *^ is -ft'- invariant, we can apply Proposition 8.1 to obtain that there 
exist H a eS(N) K such that 

[a]=k ' \y\=k+l 

By Property (S) for t he p air (N,K), for every a there is h a G i^IR^^") such that 
QH a = /i Q | E . • By Lemma 2.5 (ii), |£J and |£t>,jol are controlled by powers of |£ | on Ex>. We 



then have that 

If F a G S(N) K is the function such that QH a equals h a on it follows from ( |8.2[ ) 
that if Q = 7?.F Q for every a. This is equivalent to saying that F a (v,0,u) = H a (v,u). By 
Hadamard's lemma, 

F a (v, C, u) - H a (v, «) = E Q K j( v ' C, «) , 

3=1 

with fC, smooth. Therefore, 

[<*]=& ' [a]=jfc 3=1 ' | 7 '|=fc+l 

= ^^( & '* i *)M>«)+ E C 7 '^(t;,C,«) • 

[<*]=* ' | 7 '|=fc+l 

Notice that S" = Yl\y\=k+i C 7 '^' G 5(iV x 30) because it is the difference of two Schwartz 
functions. Since the derivatives of order up to k vanish for ( = 0, it follows from Hadamard's 
lemma that the same sum 5" can be obtained by replacing each S'y by a function S", G 
S(N x 3o ). 

Now we can undo the Fourier transform. In doing so, the monomials in ( are turned into 
derivatives in the 30- variables, and each Dj is turned into the differential operator X'^(pj), 
where the symmetrisation is taken on the direct product N = N x 3 . We denote by Dj this 
operator. 

We then have 

F =E^ a ^ + E d "' u ^ 

[a]=k | 7 '|=fc+l 

where F a satisfies the stated requirements and Uy G S(N) for every 7'. 

It only remains to replace each operator D a with the corresponding D a . In order to do 
this, it is sufficient to compare the left-invariant vector field X vo on iV corresponding to an 
element v G D with the left-invariant vector field X vo on N corresponding to the same vq. 
Clearly, the difference X Vo — X vo is a linear combination ^3=1 ^j,voi v )^zj where the £j iVo are 
linear functionals on D. Therefore, any difference D a — D a is a sum of terms, each of which 
contains at least k + 1 derivatives in the 30-variables. The corresponding term (D a — D a )F a 
is absorbed by the remainder term. □ 
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Proposition 8.3. Given F G S(N) K , there is g G S(Ifr) stzc/j i/iai, calling G G <S(A0 i/ie 
function such that QG = g\ s , then F — G G Sq(N). 



Proof. Consider the restriction of to the set in (8.1), which equals Q(1ZF) by (8.2). 
If h G S(M^ +d ») extends G(KF), let F G <S(iV) K be the function such that QF Q (£) = 

Then K{F - F ) = 0, which implies that F — F = Ei=i^ G j> with G j e 5(JV). By 
iterated application of Proposition 8.2 , we find a family {F Q } agN d 30 xN dt., 30 such that, for every 
k. 



(8.5) F = J] ^D*Fa + E W > 

[a]<fc ' M=fc+1 

and, for every «, £F a (£) = fr a (£i, with /i Q G <S(M^ +d »). 

By Whitney's extension theorem [T7] (see [2] for a proof in the Schwartz setting), there is 
a function g G <S(R d ) such that, for every a = (a/, a") G N di o x N d "- 3 o, 

We take as G the function in S(N) K such that QG = g^. We must then prove that 
G - F G S (N) K . 

Take a monomial z 13 on 3 . Given an integer k > \/3\, decompose F as in (8.5) and observe 
that 

/ (G(v,z,u) - F(v,z,u))zP dz = j (<G ^D a F )(v,z,u)z p dz , 

J} Jin r i^, a - 
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fal<fc 



since the remainder term gives integral by integration by parts. 
We set 

r*(e) = p(e)-E^(6.e.)^ 



[a]<Jfc 
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so that r k = q(g - J2[ a ]<k h DaF °) 011 

Then Lemma 7.2| gives the pointwise identity on 

[a]<k ' teT,jeZ 

We claim that k can be chosen large enough so that, undoing the Gelfand transform, the 
series 

E { G - E ^ DaF «) *3o 

t£T,j£Z [a]<k 

converges in the (S(N), |/3|)-norm (in which case it converges to G — J2[ a ]<k h.D a Fa)- 

By the continuity of Q^ 1 , there are m G N and C > 0, depending on \(3\, such that, for 
every t,j,k, 



[«]<* 



< C|| r A^t,j'IU(]K d ),m • 
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Hence we look for k such that \\rkVt,j\\s{R d ),m — 0{r for every t G T. We first do so 
with rj, replaced by the remainder sy in Taylor's formula, 



|a[<Jfe' 

Then, for 7 G N d with I7I < m and for every M G N, 

l^(^^-)(o| < c m , M {\ + 2-^')(i + ia~ M (ia + i^oi) fc,+1_m , 

where m only depends on m. 

From Lemma [2.5| we obtain that there are positive exponents a, b, c, a', 6', c' such that, on 



->T>, 



(8.6) for |f I small: 



Idol ~ l£»| a 
l£e,3o I l&l%ol 



for |f | large: 




< l^ol"' 
<l^| 6 'l4ol 



Since the inequalities of Lemma 2.5 remain valid in a Z}(£)-invariant neighbourhood of 
Ex>, we may assume that (8.6) hold uniformly on the support of each Sk>r] t j. 

Since there are r, r' > such that, on the support of rj t j, |f 3o | < r TJ if j < 0, and |f 3o | < r r 5 
if j > 0, we can choose fc' such that ||sfc'??tj||s(Rd) im = 0(r~^). 

Finally, we choose k = max{[a] : |a| < k'}. Since k > k', 

\a\>k' ,[a]<k 

The estimates on ||(r fc — Sfc')?7ij|| < s(a d ),m are basically the same. 

Now, — X][ a ]<fc h.D a F^\ * i0 ipt,j G $o{N) K , because its 30-Fourier transform vanishes 

for ( G 30 close to the origin. To conclude, observe that integration against z@ is a continuous 
operation in the (S(N), |/3|)-norm. □ 

Corollary 8.4. Let (N,K) be one of the pairs in Table^ and assume that Property (S) 
holds on all its proper quotient pairs. Then Property (S) holds on (N,K). 



Proof. Given F G S(N) K , let G be as in Proposition 8.3 By Proposition 7.1, Q(F — G) 
admits a Schwartz extension h. Then g + h is a Schwartz extension o f QF . This shows that 
the map Q^ 1 : <S(Ed) — S(N) K , which is continuous by Theorem is also surjective. 
By the open mapping theorem for Frechet spaces |20j , it is an isomorphism. □ 



An inductive application of Corollary 8.4 gives us Theorem 1.2 for pairs in the first two 
blocks of Table [I] In fact, the classification of quotient pairs given in the appendix (Sec- 
tion 10) shows that this set of pairs is essentially self-contained, in the sense of Remark 5.1 



The lowest-rank summands that need to be considered in order to start the induction are 
the following: 

• line 1: the trivial pair (R, {1}) and (Hi, SO2); 

• line 2: (#i,Ui) = (ifi,S0 2 ); 

• line 3: the quaternionic Heisenberg group with Lie algebra EI © Imi, and with Sp x 
acting nontrivially only on D = HI; 
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• lines 4, 5, 6: (C,{1», (#i,Ui), and (R,{1», respectively. 
Since in all these cases Property (S) is either trivial or proved in [TJ, we can state the 



following restricted version of Theorem 1.2 



Corollary 8.5. Property (S) holds for the pairs in the first two blocks of Table^ 



9. The third block of Table Q] 



The pairs in the third block of Table [T] must be treated separately because their quotient 



pairs (cf. Table pi) do not satisfy Vinberg's condition. In order to apply Corollary 8.4 



we 



need to prove Property (S) for each of the quotient pairs listed in Sections 10.6, 10.7, 10.8 
We call them "first-generation" quotient pairs. 

Let (N, K) be one of these quotient pairs. The group N is a. direct product, N\ x N 2 , with 
xij — 0j©3j, j = 1, 2, and Di = t>2- The group K is the direct product of three simple factors, 
K — K\ x K12 x K 2 , where the subscript indicates on which of the two factors of N the given 
factor of K acts nontrivially. In particular, K\ 2 acts by its defining representation on both 
K>\ and D 2 , and trivially on fa and Moreover, when Kj = Sp x , it acts on ij = ImH = spi 
by the adjoint representation. 

To prove Property (S), we adapt the paradigm used so far, and this involves the following 
steps: 

(i) identify a critical subset of the spectrum such that its complement can be locally 
identified with spectra of certain "second-generation" quotient pairs; 

(ii) prove that Property (S) holds for each second- generation quotient paiiQ 

(iii) deduce the analogue of Proposition 7. 1 for the first-generation quotient pairs; 

(iv) find a group N such that the spectrum of (N, K) can be identified with and prove 
a corresponding form of Proposition 8.2| 

The first-generation quotient pairs are shown at lines (a), (b), (c) of Table [2] with their 
fundamental invariants, and the second-generation pairs are shown at lines (a'), (b'), (c'). 
When i>i,i>2 have several components, they must be understood as row- vectors. At lines 
a, b, a', f, the vectors vi,v 2 are complex, while they are quaternionic at lines c and c'. At 
lines c, c', the factor Ui in K acts by (left) scalar multiplication by e td on t?i = W 1 . 

For a better organisation of this material, we start with the second-generation quotient 
pairs, prove Property (S) for them, and then pass to the first generation. 



9.1. Second-generation quotient pairs. 

The pairs involved at this stage, listed in the second block of Table [2j will be the ones ob- 
tained from the first-generation pairs by factoring out of n the centre of the second summand. 
The group K remains unchanged. 

In this subsection, we prove the following statement. 



This will require a repetition of the paradigm involving some "third-and-last-generation" quotient pairs, 
for which Property (S) is known to hold. 
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Ki x K lt 2 x K 2 


n = ni ffi ti2 


p., 


Pt, = P 0l U Pt> 2 U P01.B2 


Pt.,3 


a 


Ul X SU„ X Ul 




21 , ^2 


|Ul| , \V2\ 

l?>i 1>X I 2 Cif T) > 91 




b 


Ui x Sp n x Ui 
in > 2) 


f)2n ffi f)2n 


21 , 2 2 


H a .N a 

|viv|| 2 

|«lJ*«2| 2 




c 


Ui x S Pn x Spi 


(HI™ ffi R) ffi (H n ffilmH) 


21 , |2 2 | 2 


|viV2*| 2 (if n > 2) 


Re (i(fifa)2 ! a(w2i'i)) 




Ki x Ki,* x K 2 


Ul ffi t»2 


Pll 


Po 




a' 


Ui x SU„ x Ui 


l)„ffiC n 


21 


same as line a 




b' 


Ui x S Pn x Ui 
(n > 2) 


ban ffi C 2n 


21 


same as line b 




c' 


Ui x Sp n x Sp x 


(e n © r) © bp 


21 


same as line c 





Table 2. First- and second-generation quotient pairs for lines 7-10 of Table 1 



Proposition 9.1. The nilpotent Gelfand pairs (Ni x t> 2 , K) at lines a', b', d of Table^ satisfy 
Property (S). 

As it was mentioned before, the proof involves the introduction of third-generation quotient 
pairs, which are constructed in analogy with Section [5j Here we let 31 © o 2 , the centre of 
rii ffi D 2 , play the role that was of 3 © 30 in Section |5j 

Given t 7^ in t) 2 , we factor out the tangent space t 1 - from D 2 , i.e. the tangent space to 
the if-orbit in t) 2 . The resulting quotient Lie algebra, n t = n± ffilR, is either t) n ffilR (line a'), 
or f) 2 n © K (lines b', c'), with 

{Ui x U n _i line a' 

U? x Sp n _! line V 

Ui x (Spj x Sp„_ 1 ) line c' , 

where the first two actions are related to decompositions of Di as C n = C ffi C™^ 1 , C 2n = 
C 2 © C 2n " 2 , and HP = HI © I"" 1 respectively. 

Lemma 9.2. The third- generation quotient pairs (N t ,K t ) satisfy Property (S). 



Proof. Since N t is a Heisenberg group, the statement follows from [2] and Proposition 3.3 □ 

Let (N% x t) 2 , X) be one of the second-generation pairs, and (N\ x IR, K t ) the corresponding 
third-generation pair. 

A homogeneous Hilbert basis p t on is given by the norm squared on the irreducible 
components of t>i and by the two coordinate functions on 31 and M. We let D t be the system 
of their symmetrisations on N\ x R. 

Let T> be the systems of symmetrisations on N\ x v 2 of the invariants in Table [2j To points 
£ G Tj V we assign coordinates (£ 3l ,£t>i, £t> 2 >£t>i,t> 2 )- 

Let Ex>, resp. £x> t , the two embedded Gelfand spectra. In analogy with Lemma 2.4 and 
Proposition 6.1, we have surjections 

Y, Vt AE c ^p 02 (D 2 ) = [0,+oo). 
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We set 

± v = IL-\0) = {£ : £„ 2 = = 0} C S c . 

This set represents the spherical functions that do not depend on the ^-variable. Hence 
it is naturally identified with the spectrum of (Ni, K). 

Away from A _1 (£x>), A is a homeomorphism, and both A and A -1 are restrictions of 
smooth maps (cf. Propositions 6.4 and 6.5). Therefore, setting 

S (N 1 xv 2 ) = {FeS(N 1 xv 2 ): [ F(v 1 ,z 1 ,v 2 )dv 2 = V(t>i,*i)}, 

the spherical transform QF of any F G Sq(Ni x ti 2 ) K can be extended to a Schwartz function 
vanishing with all derivatives where £„ 2 = (cf. Proposition 7.1). 

The next task is then to establish the following analogue of Proposition |8]2 for the second- 
generation pairs. The different formulation that we give below is required by the fact that, 
due to the presence of the abelian factor t) 2 , t>i does not generate n. This implies that smooth 
multipliers of operators in A^(rii) cannot have Schwartz kernels on iVi x t>2 • 

Let p' = p V2 U p „ 2 C p, V = X' NlXV2 (p') and d' its cardinality. For a G N d ', by [a] we 
denote the order of differentiation in the D 2 -variables of the operator D a , as a monomial in 
the elements of T>' . 

The analogue of Proposition is as follows. 

Proposition 9.3. Let F G S{N\ x t> 2 ) , and assume that 

F(v t ,Zi,v 2 ) = ^2 d2 2 R~f(v 1 ,z 1 ,v 2 ) , 
h\=k 

with R 1 G S(Ni x t) 2 ) for every 7. Let also \f G S(t3 2 ) K , with Fourier transform ^ equal to 
1 on a neighbourhood of the origin. 

Then, for every a with [a] = k, there exists a function F a G S(N\) K such that 

F(v 1 ,z 1 ,v 2 ) = ^ i D a (F a (v 1 ,z 1 Mv 2 )) + dt R 'M,zi,v 2 ) , 

[a]=k ' | 7 '|=fc+l 

with R ' , G S(Ni x (j 2 ) for every 7'. 

Proof. As in the proof of Proposition |8.2 we begin by taking partial Fourier transforms in 
the t> 2 -variables (denoted by F, i? 7 etc.). We have 

F(vi, z ± , w 2 ) = i k ^2 W 2^j( V ^ z ^ w z) 
| 7 |=fc 



l7l=* 



+ i k W 2*M (Rj( v u zi, w 2 ) - Rj(v u z u 0)) 

\l\=k 

+ i k wl(l - ^(w^R^vu Zl ,w 2 ) . 



\l\=k 



37 



The functions $f(w 2 ) (i2 7 (i>i, z\ 1 w 2 )—Rj(vi, z\ 1 0)) and (l— ^(w 2 ))Rj(v 1, zi, ^2) are Schwartz 
and vanish for u>2 = 0. Applying Hadamard's lemma, we have 

(9.2) F(v 1 ,z 1) W2) = i k $(w 2 )^w2R Y {v 1} z 1 ,0)+i h+1 w^R'^{v u z u w 2 ) , 

\l\=k \-y'\=k+l 

with R' y £ S(Ni x t) 2 ). We set 

(7(^1, zi, iu 2 ) = ^ W2-R 7 (^i, Zi, 0) , 
| 7 |=fc 



(9.3) 
and 



D j = X' Nl (p j ) G B(JVi) <g> 7>(t> 2 ) 



so that 
(9.4) 

for F e 5(JVi x d 2 ). 

We prove the analogue of Proposition 8.l[ with Dj replaced by Dj. 

By Lemma 4.1 of [TU], the function G in ( |9.3[ ) can be expanded as a finite sum, 

(9.5) 



G(v 1 ,z 1 ,w 2 ) =^2\w 2 \ 2m p a {v 1 ,w 2 )g a (v 1 ,z 1 ) , 



where the p a are products of elements of p Di D2 of degree at least k — 2m in w 2 and the g a 
are in S(H n ) K . Since the elements in p D D2 have degree 2 in w 2 , we may assume that the 
integer k in (9.3) is evennand that 2\a\ + 2m = k in (9.5). 



We need at this point the following statement, whose proof will take the second part of 
this subsection. 



Lemma 9.4. Given 

(9.6) G( Vl , z h w 2 ) = z x ) G (S(H n ) ® P 2fe (t> 2 )) 



A' 



| 7 |=2fc 



then 
(9.7) 



F2 



2(k-\*\)j) a H 



G(v 1 ,z 1 ,w 2 ) = ^2 

\a\<k 

- — a — - /-v -■ 

where D = FJ^ 

Taking Lemma 9.4 for granted, we insert ( |9.7 ) into (9.2) and undo the Fourier transform 
to obtain the conclusion of Proposition |9.3| □ 



iDj&i'fa, ) ' and H a G S(H n ) K for every a. 



Before giving the proof of Lemma 9.4, we recall some notation from [10] and quote two 
preliminary results. 

If V is a real vector space, 1-L k {y) denotes the subspace of V k (V) consisting of harmonic 
polynomials, i.e., orthogonal to |i>| 2 . If V is also a complex space, we refer to the holomorphic- 
antiholomorphic bigrading by replacing the single exponent A; by a double exponent. 

The first of the two statements we alluded to can be found, for example, in 



In fact, if (9.3) holds with k odd, it holds as well with k + 1 instead of k. 



:;<s 



Lemma 9.5. Under the action ofSp n , % m > m (C 2n ) decomposes into irreducibles as 

m 

(9.8) H m,m^ = v mti , V m>i = R(2{m - i)w x + iw 2 ) , 

i=0 

where w\ and w 2 denote, respectively, the highest weights of the defining representation and 
of the adjoint representation o/Sp n . 

Under the action ofSp 1 x Sp n; "H 2m (IHI n ) decomposes into irreducibles as 

m 

(9.9) H 2m {W l ) = Y,W m , , 

i=0 

with W m: i = 5 ,2 ( m ~ t ) Vm^i, Si denoting the (j + 1)- dimensional irreducible representation 
o/Sp 1 . In particular, Sp x acts trivially on W m>m = V m ,m- 

The second statement is extracted from Sections 4 and 5 of |10|. 



Lemma 9.6. Let ~N\ = H n , K C U n , and V s,0 (tii) irreducible with respect to K for every s. 
Let Pvi,v 2 be the polynomial in (9.11), with V\, V 2 equivalent, irreducible representation 

spaces ofK, with V x C H m > m {y> x ), V 2 C V k {y> 2 ). Set M Vl y 2 = X' Nl {p Vl y 2 ) e (B(^) g> V 2 ) K . 
Assume that 

(i) V s,0 (t>i) is contained, as a representation space of K , inside "P s '°(di) Cg> V 2 if and only 
if s >m, and in this case with multiplicity one; 

(ii) dTi(M Vl y 2 ) 7^ on V s ' ($i) for every s > m. 

Then, given g G S(H n ) K , there exists H G S(H n ) K such that p Vl y 2 g = M Vl y 2 H. 



Proof of Lemma 9.4 In the case n = 1 (which is admitted at lines a' and d) there is 

nothing to prove, because p' = {|w2| 2 } and A 7 ( | if 2 1 2 ) = \w 2 \ 2 . Hence (9.7) and (9.3) coincide. 
Therefore, we assume that n > 2. Following the procedure of [TO] , we want to split the factor 
V 2k {y> 2 ) in the tensor product into irreducible components, and select those components V 
for whic h (S (H n ) <g> V) is nontrivial. 

From (9.5), we isolate a single summand. Disregarding the terms containing a positive 
power of \w 2 1 2 , which can be dealt with by induction, we may then assume that 

(9.10) G = p a (v 1 ,w 2 )g a (v l ,z l ) , 
with k — \a\. Now, 

P a e (p 2fc (t>o ® r 2k (t> 2 )) K = {vil^M^^n*^!) ®u 2j (t> 2 )) K . 

i ,j<k 

For i 7^ j, any polynomial in (V 2l (&i) <S> V 2 i(b 2 )) K must be divisible by a power of \vi\ 2 

or \w 2 \ 2 . Hence {H 2l {y>i) ®% 2j {p 2 )) K is trivial for i ^ j. 

As in [TO], given V\ C V{t>\), V 2 C V(t> 2 ), K-invariant, irreducible, and equivalent, we set 

(9.11) p Vl y 2 ( Vl ,w 2 ) = ^aeivjbtiwz) G (V x ® V 2 )* , 
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where {ae}, {be} are equivalent orthonormal bases of V\ and V 2 respectively. We say that 



Pv lt v 2 * s an irreducible mixed invariant on Oi 
decomposed finite sum 



X) 2 . For every a with at = k, p a can be 



|2(fc— rrij) I 12(^—7^^) 



(9.12) 

with C n 2m i(t> 2 ), V 2>j C H 2m '(t> 2 ), and the function G in gig) 
We can then further restrict the study of (9.5) to the case 

(9.13) G =Pv 1 ,v a (ui,iW2)fl , («i,«i) , 

with Vi C T-L 2m (\Ji), V 2 C H 2m (t)2), equivalent if-invariant, irreducible subspaces. 

Obviously, if V\ and V 2 are as above, then ifi acts trivially on Vi and if 2 acts trivially on 
V 2 . In particular, if ifj = Ui, in which case 0, has a if- invariant complex structure, we have 

Vi c n m ' m (^). 

that the irreducible mixed invariants m ("% 2m ' 



It follows from Lemma 



"H 2m (o 2 )) correspond to the following pairs of subspaces 



9.5 



(9.14) 



line a' : 
line b' : 
line c' : 



2 
D 2 



C n , 

C 2n 
W 1 , 



^1 
, K 



v 2 
v 2 
v 2 



n 

v n 

Vrr 



m,m . 



m 



Suppose therefore that G G (S(H n ) <g> V) is the function in (9.13), with V 
V mi {x> 2 ), or V mm {K> 2 ), depending on the case. 



W 



J 2), 



Setting My = X' N (p 



we prove that G = MyH for some H G S(Nx) . This 



will give the conclusion, for the following reason. The operator D 



v 



A TV 



Nixt> 2 \PV(t>i),V(t>2)j 



3(Ni x 02)^ is a polynomial in the elements of T> = T>' U {L,z _1 T}, where L and T are, 
respectively, the sublaplacian and the central derivative on N%. Hence My = Dy is a 
polynomial in |u>2| 2 , L, T and the Dj in (9.4). In each monomial, the powers of L and T can 
be incorporated in the function H, leading to a sum of the form (9.7). 

Let then 7r be a Bargmann representation of N\ = H n , acting on the Fock space J-'(t)i). By 
a we denote the natural representation of if (effectively of K\ x if 1,2) on .F(t)i), such that 
cr(k) intertwines tc with -r ok for k G if. Since if 12 is either the unitary or the symplectic 
group on 0i, the subspaces "P s '°(t)i), s G N, are the irreducible components of a. 



m our cases. By Lemma 4.4 of [10J, V s, ° is 
V 2 if and only if the same is true for V 2 inside V s ' ® V 0,s = V s ' s , and 



i=0 



2(s 0%*.^ condition (i) is easily 



We verify the hypotheses of Lemma 9.6 
contained inside V s, ° 

the two multiplicities are the same. Since V s,s = 
verified on the basis of (9.8) and (9.9). 

Condition (ii) for the pairs at line a' is already contained in Proposition 4.10 of [10J. We 
consider then the pairs of line b', with V 2 = V m ^. 

We first replace the elements of p 0l>02 in Table [2] with py li0 (tu),Vi,o(t>2) and Pvi,i(oi),vi, 1(02)5 
which we simply write py 10 and py 11 - 

We then expand pv 10 and py x 1 in terms of equivalent orthonormal bases {£o,j} and {£i,j>} 
of Vifi and Vi,i respectively, as in ( |9. 11 ). Then, 

Bpanc^-o,^,!}^ = H 1 ' 1 = {vxCvl : C G s[ 2 „(C)} . 
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For C G sl 2n (C), define t c {v) = vCv* on C 2n , and L c = X' Nl (£ c )- According to [TQl 
Lemma 4.11], the restriction of the operators dir(Lc) to V s '°(tii) coincides, up to a nonzero 
scalar factor depending on ir, with da(C), the differential of the natural action of SL^C). 

We choose the orthonormal bases so that £1,0 = £c , £1,1 = £d are lowest-weight vectors, 
respectively in V^o of weight —2wi, and in V^i of weight — vd%. Given D = ^2 r D r (g> 
q r G D(iVi) <g> V k (t> 2 ) and £ G V k (t> 2 ), the notation (D,£) stands for 52r(q r ,£)D r G D(JVi), 
and similarly with differential operators replaced by their images ir(D r ) in the Bargmann 
representations, or by polynomials on t>i . 

We claim that 

(i) for every m,i, i < m, ^™ ~^i,i £ V m ,i] 

(ii) (M mt i, £^0^1,1) equals, up to a nonzero scalar, (D^D^ ,1™^^)] 

(iii) for s > 771, (^(M^J^T^) G £(P s '°(0i)) is nonzero^ 

To prove (i), notice first that each £™q % £\ x is a lowest-weight vector vector and its weight, 
— 2(m — i)cci — IW2, does not appear among the lowest- weights in the lower-degree har- 
monic spaces. Therefore Piq 1 £\ 1 G H m ' m . Being a lower- weight vector means that the 
one-dimensional spaces that it generates is invariant under the action of a Borel subalgebra 



of sl 2n - Then the same is true for £™ % £\ v Since, by (9.8), V m % is the only irreducible sub- 



'1,0 H,i- 

space of % m ' m with the highest weight 2(m — i)w\ + izu 2 , P\$ l £\ 1 must be the lowest-weight 
vector in V m ^. 

To prove (ii), we first observe that the statement is true with M m i and D™q 1 D\ x replaced 
by pv m ,i and Py~*p Vl 1 respectively. To see this, it suffices to replace Py~^p Vl x with its 
component q m ^ in 'H m ' m (Vi) ® H m,m (v 2 ), which is a linear combination of the py m . with 



j = 0, . . . ,m. On the other hand, comparing the expansions (9.11) of py mi , Pvi, Q and pv 1A 



with bases consisting of weight vectors, we must have a triangular set of linear relations 

Qm,m—l b m m—lPV m ,m ^"m—lPV m ,rn—l 



just because the weights 2(m — j)w\ + jto 2 are decreasing in j (i.e., 2wi — w 2 is a positive 
root). The fact that the homogeneous Hilbert bases in Table [2] is free [16] implies that each 
coefficient aj is nonzero. It is then clear that 

K>v ltl ,<?^4,i> = «i(pv m>i ,^i,i> • 

It is now sufficient to apply the symmetrisation to both sides and observe that A^ is 
multiplicative up to terms containing factors in 31, which will necessarily have lower degrees 
in the vector fields in t»i , and hence in t) 2 . 



The scalar product is taken in the second factor, V m j, of the tensor product. 
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To prove (iii), we use the identity 

(d7r(M m>i ),£T,i%,i) = (<fcr(i)f^).^i> 
= drr(L^F C2 ) 

In an appropriate coordinate system (d, . . . , ( 2n ) on (recall that n > 2), we can take 

dcr(Ci) = Cn+i% , rfcr(C 2 ) = C«+2<% - Cn+i<9 ?2 , 

so that da(Ci) m ~ l d<j(C 2 ) 1 does not vanish on V s ' for s > m (e.g., check the action on 
Finally, the case of line c' leads to essentially the same situation, with i = m. This 



concludes the proof of Lemma 9.4 □ 



9.2. First-generation quotient pairs: proof of Property (S) for pairs at lines a and 
b. 

As usual, we denote by D the homogeneous basis of D(A^) K obtained from the invariants 
in Table |5J with D\ = \' N (zi), D 2 = \' N (z 2 ). Given a point £ G with ^ (a regular 
point), we set (£i,£2) = Xug, with ug = (cos 9, sin 9). Then the corresponding spherical 
function ip^ factors to the Heisenberg group Ng with Lie algebra rig = (Di © D 2 ) © — 
(Oi © o 2 ) © ^u . 

Notice that the pairs (Ng, K) are all isomorphic to the Heisenberg pair (H m , K) with 
m = dimc(tJi ©1)2). The map (v\,v 2 ,t) 1 — > (v\,v 2 ,tug) is an isomorphism from H m to Ng. 

On H m we keep the invariants in p„ of Table |2j adding to them the coordinate function 
t on the centre. Calling T>h the resulting system of differential operators, the point of ^v H 
corresponding to yjg has coordinate^! 

(\ J?_ & & ) . 

V ' cos 9 ' sin 6 1 ' sin 6 1 cos 6* ' sin 6* cos 6> / 
For F G S(JV)* = ^(^F)(^(0), with 



lZgF(v 1 ,v 2 ,t) = / F(v 1 ,v 2 ,tug +7T/2 )dt e S(R n 



\K 



Let 77(0) be a C°° function supported on a compact interval / C (—11,71) \ {0,±|} 
and equal to 1 on a neighbourhood of a point 9q. Then the function F#(v\,v 2 ,t,s) = 
F e l (ri(9)1Zg(vi,v 2 ,t))(s) is if-invariant and Schwartz on iV# = H m x K. With respect to 
the system T># = T>h U the spherical transform of _F# on S^* = x R is 

e # (F*)(*(o,f)=feM($(0)=TO • 



By Proposition 3.3 and [2], G#(F#) (^(£), 9j admits a Schwartz extension from S^* to 
the ambient space. Since the map £ i — )■ (^(^), 9) is a diffeomorphism on the set of £ with 
A > and # G /, we conclude that (?.F admits a smooth extension to any set of £ G M d 



'The coordinate £g is only present in the pairs of line b, n > 2. 
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(d = 5 or 6) with (^1,^2) varying in a compact set with £i£ 2 7^ 0, with derivatives of any 
order decaying rapidly in the remaining variables. 

Denote by S fi(N), resp. S (N), the subspaces of S(N) whose elements have vanishing 
moments of any order in Z\ and z 2 separately, i.e. 



F(vi,v 2 ,z 1 ,z 2 )z 1 dz 1 = / F(v 1 ,v 2} z 1 ,z 2 )z 2 dz 2 = , 

for every k and every choice of the non- integrated variables, resp. vanishing moments of any 
order in z\ and z 2 jointly, i.e. 

/ F(vi, v 2 , z x , z 2 )z kl z 2 2 dz\ dz 2 = , 

for every k±, k 2 and every choice of vi,v 2 . Clearly, So,o(N) C Sq(N). 

Then Sq j q(N) k consists of the functions F G S(N) K with QF vanishing of infinite order 
on the singular set (where £1^2 = 0), and Sq(N) k of the functions F G S(N) K with QF 
vanishing of infinite order where £1 = £2 = 0. 

Since the derivatives of ^ grow at most polynomially as £1 or £ 2 tends to 0, we can say, 



using an appropriate partition of unity on the regular set as in the proof of Proposition |7.1 
that, for every F G iSo,o(-^0 X > QF admits a Schwartz extension to M, d . 



We invoke now the following analogue of Proposition 8.2 

Lemma 9.7. 

(i) Let F G S(N) K , and assume that 

F(vi,v 2 ,z 1 ,z 2 ) = d k ^R k (v u v 2 , z 1 ,z 2 ) , 
with Rk G S(N) K . Then there is a function F k G S(N) K such that QF k does not 



depend on £ 2 and 



d k Z2 F k + d k z ^R k+l 



with R k G S(N) K . 
(ii) Let F G S(N) K , and assume that 

F(v 1 ,v 2 ,z l ,z 2 ) = ,V 2 ,Zi,Z 2 ) , 

i+j=k 

with Ri j G S(N) K for every i,j. Then there are functions F(j G S(N) K , with 
i + j = k with spherical transforms QF it j that do not depend on £1,^2 and 



r+s=fc+l 

with R rtS G S(N) K for every r, s. 

This is essentially Geller's lemma as cited in p], cf. Theorem 2.2 and Lemma 5.2 therein. 
We sketch the proof for completeness. 

Proof. Let R k (vx, v 2 , z\) = f R k (vi, v 2 , z\, z 2 ) dz 2 . Then R\ G S{N\ x t^)^ and the spherical 
transform Q 'R b k ( ^i, £ 3 , • • ■ ) - for the pair (iVx © D 2 , K) - coincides with QR k (£i, 0, £3, • • • ). By 
Proposition 9.1, Q'R\ extends to a Schwartz function ^(^1,^3, • • ■ )• 
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Setting g k (£i, £2, £3, ■ ■ ■ ) = <7fc(6i, 63> ' ' ' )> t ne rest ricti on of to the Gelfand spectrum 
E© for the pair (N,K) is in <S(Ex>). By Theorem ll.ll there is Gk G S(N) K such that 
^Gfc = . Then — G^) vanishes for £ 2 — 0, and this implies, by Hadamard's lemma, 
that Rk — Gk is the ^-derivative of a i^-invariant Schwartz function Rk+x- 

This proves (i). The proof of (ii) is similar. □ 



Using part (i) of Lemma 9.7, we can repeat the proof of Proposition 8.3 to prove that the 



spherical transforms of functions in Sq(N) k admit a Schwartz extension, and next, using 
part (ii), that the same is true for general functions in S(N) K . 



9.3. First-generation quotient pairs: proof of Property (S) for pairs at line c. 

In this last case, K acts nontrivially on one component, 32, of the centre. As in Section [5j 
we reduce part of the proof to a quotient pair (N', K'), factoring 32 by (any) two-dimensional 
subspace and taking K' as the stabilizer of the factored subspace in K. This quotient pair 
is isomorphic to the pair at line b with the same n, only with K 2 = U\ x 7L%. For simplicity, 
we factor out the subspace orthogonal to i in ImE 

Denoting by pi, ■ ■ ■ ,p&, resp. p[, . . . ,p' 6 , the invariants in Table |2]on n and n' respectively 
(in the same order and with p' 2 = z\ to take into account the extra Z2), we have the following 
relations: 

(9-15) Pj Lf = p'j , (1 < j < 5) , p 6L , = v^Oe - Ps) ■ 

Denoting by T>, T>' the symmetrisations of the systems {pj} and {p'j} respectively, and 
by Exi, Ex>/ the corresponding spectra, we have a continuous surjection of E©/ onto E© (cf. 



Proposition 6.1) which is a diffeomorphism from the complement of the singular set £ 2 = 



in E75/ to the complement of the singular set £2 — in E© (cf. Proposition 6.5). 



By Proposition 3.1, the pair (N',K') satisfies Property (S), hence a repetition of the 
arguments used in Section u\ implies that the spherical transforms of functions F E Sq(N) k 



admit a Schwartz extension to IR 6 , where Sq(N) is the space of Schwartz functions with 

F(v 1, t> 2 , Zi, Z2)z 2 dz 2 = 



32 

for every a e N 3 and every Vi,v 2 ,Zi. 

At this point, we are back at the situation of Section |8l with 31 and 32 playing the role 
that was, respectively, of 3 and 3 . Setting, as in Section ]8| N — Ni x D 2 , we observe that, 
once again, the singular part of Ex>, 

SD = {(es P :6 = o}, 

is naturally identified with the spectrum E^ of the pair (N,K), i.e., the pair at line d of 
Table [2J with t> constructed from the invariants in the same table. 

Following again the procedure of Section |8l we reduce matters to proving an adapted 



version of Proposition 8.1 

We set Dj = X'^ipj), where pi, . . . ,p$ are the invariants at line c of Table 
only p 2 = \z 2 \ 2 and p 6 = Re (i{viv 2 )z 2 (v 2 vl)) contain the variable z 2 . 



Notice that 
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Lemma 9.8. Let G be a K -invariant function on N x $ 2 of the form 

(9.16) G(vi,v 2 ,z 1 ,z 2 ) = z 2 G a (vi,v 2 ,Zi) , 

\a\=k 

with G a G S(N) and z 2 G 32 = ImH. There exist functions Hj jTn G S(N) K , for 2j + m = k, 
such that 

(9.17) G= lLH^ E ^- 



v.rrv. 

2j+m=k J 



to 



Once this is provecp] we can repeat the proofs of Proposition 8J2 and Proposition 8.3 
obta in t he conclusion. To prove the lemma, we must adapt part of the proof of Proposi- 
given in jTO], Section 5] to the new situation where N has an abelian factor. 



tion 



Proof of Lemma \9. S\ Repeating arguments used before, it is easy to see that, for every m, 
there is a unique i^-invariant polynomial p m on n, belonging to (% m ' m (t)i) © 7-L 2m (& 2 ) © 

U m (} 2 )) K , and that the (U m > m (t)i) ©H 2m (o 2 ) © H m (3 2 ))-component of p% is nonzero. Then, 
every i^-invariant polynomial p on n can be uniquely expressed as a finite sum 



P=2_^\Z2\ ' ''PmQi,m{Vl,V 2 ,Z 1 ) 
i,m 

with q i>m eV(n) K . 



We decompose G G (S(N) © P fc (3 2 ))^ as 

2,^1,^2) = l^l^^^i,^, 21,22) , 

2j'+m=fc 

with G m G («S(iV) © n m (^ 2 )) K . It will suffice to show that 
(9.18) G m = M m H m , 

where M m = X^(p m ) G (B(N) © H" 1 ^))^ and # m G S(JV)* 

Since iV is the product of the Heisenberg group N% and the abelian group V 2 , the infinite 
dimensional irreducible unitary representations of N are the tensor products 7i\ © where 
7Ta is a Bargmann representation of N\ on .F(t>i), and % w is the character e 1 ^'^ of i> 2 , with 



wG D2 (cf. Lemma 2.2) 



In the generic case u 7^ 0, the stabiliser of 7Ta®Xw m ^ is isomorphic to Ui xSp x xSp n _ l5 
with actions on Di = Mu © (Hw)- 1 = H © H n ~ 1 and on 32 = ImH given by 

(v, v') 1 — ► e ie (vk-\ v'k'" 1 ) for (v, i)')eH0 H™ -1 , 
z 2 1 — > kz 2 k~ x for z 2 G ImH . 



Ui x Spi x x 3 (e ie , k, k' 



Then the ^-invariant irreducible subspaces of ^(f 1) are the tensor products 
V Sl '°(M) © P S2 '°(H n - 1 ). By [ID1 Proposition 4.5], tt a © x w (G) can be nonzero only if V? u 
is contained, as a representation space of K u , in © W n {] )2 ). This is equivalent to 



Si 

S2 



11 Once again, we do not need to worry about control of the Schwartz norms of Hj m in terms of Schwartz 
norms of the G a . 
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saying that V Sl,0 (M) is contained (and with the same multiplicity) in V Sl,0 (M) <S> T-L m {l2) as 
a representation space of Ui x Sp x . By [TUl Proposition 4.6], this happens if and only if 
Si > m, and with multiplicity 1. 

We show next that, for si > m, d{jz\ <S> Xw){M m ) is nonzero. Observe that 

d{it\ <S> Xu){M m ) = i m dix x {\' Nl {pm{vi,u, Zi))) , 

and that 

Pm (-,u,-)e (H m > m (xn)®U m fa)) Ku = (^ m ' m (e w )®^ m ( 3l )) UlxSpi . 

This is one of the cases considered in the proof of [TUl Proposition 4.10], and we conclude 
that d{-K X ® X u>){M m ) ^0. 

So we are in the following situation: the spectrum E-g, of (N, K) is the closure in IR 4 of 
the set of points £ with 

6 = A, 6 = 2|A|( Sl + s 2 +n) , & = M 2 > U = 2\u\ 2 \\\( Sl - s 2 -n + 2) . 
It is convenient to replace £4 by 

e 4 = e 4 + 6e 3 = 4|a;| 2 |A|( Sl + i) , 

and set £' = (£1, £2, £3, £4)- Correspondingly, we replace _D 4 € P by -D4 = Z) 4 + -D2-D3 and 
call X>' the resulting system of differential operators. 

The operator U m = M m M m is in 3(N) K , so there is a polynomial u rn such that U m = 
u m (V'), and 

Then w m vanishes on the set i£ m C S^, of points £' corresponding to si = 0, . . . , m — 1, i.e. 
where £ 4 = j|£i|£3, with j — 1, . . . , m. Repeating the proof of [TOj, Lemma 5.1], we conclude 
that 

m m 

= c m - m 3 ) Hie + m 3 ) , 

3=1 3=1 

with c m 7^ 0. 

Consider now M^G m G S(N) K . Then £(M^G m ) vanishes on i? m . We can then apply 
[TUt Proposition 5.2] to conclude that Q(M m G m ) admits an extension to IR 4 of the form u m ip 
with tj) Schwartz. If H m e S(N) K is such that QH m = ^Ie-,' we have M^G m = U m H m . 
Repeating the conclusion of [TpJ, Section 5], we then have 

{d(7i x ® x«)(M m ))* (tt a ® X u)(M m H m - G) = . 

The conclusion follows by the invertibility of rf(7TA <g) Xcj)(M m ) on the spaces s where 
(tt\ £S> Xuj)(M m H m — G) might not vanish. □ 
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10. Appendix: quotient pairs generated by pairs in Table [1] 



Let I p (resp P ) be the identity (resp. zero) p x p matrix and 

J p := diag( J, J) ■ 

p times 

The Lie bracket [ , ] is understood as a map from o x D to 3, elements of M. n , C n etc. as 
column vectors. 



10.1. The pair (R n © so n , SO„). 
The Lie bracket is 

r '1 ^ / t 1 1 t \ 
[V, V \ = -{V V — V V) . 

Up to conjugation by an element of K, we may assume that t has the form 

t = diag(*iJ pl , . . .,t k J Pk ,0 q ) , 
with 2pi + • • • + 2pk + q = n and ti 7^ tj 7^ for every i j. Then we have 

fi = U Pl x---xU ft x S0 9 , 
3t = \Xp! © • • • © u Pfc © so 9 , 

n t = (C P1 © UpJ © • • • © (C P1 © UpJ © (R q © 50 9 ) . 

10.2. The pair (C n ©u„,U n ). 
The Lie bracket is 

|t>, f J = - (vv — v v ) . 

For 

t = diag(iti/ Pl , . . .,it k I Pk ) , 
with pi + • • • + pk = n and tj 7^ t,- for every i 7^ j, we have 

#t = U P1 x • • • x U Ph , 

It = U P! © " " " © U pk , 

nt = (C P1 © u Pl ) © • • • © (C P1 © u p J . 
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10.3. The pair (H n ffi (HS 2 M n ffi ImH), Sp n ). 
The Lie bracket is 

[v, v'\ = - (viv'* — v'iv* tr (viv'* — v'iv*)I^\ © Im (v*v) 

= [-(viv'* - v'iv*) - -lmi(v*v')I n ^ ®lm(v*v') , 

where Iim denotes the i-component of the argument. 
For 

t = diag(ti/ pi , . . . , t k I Pk ) © (mi + u 2 j + u 3 k) , 
with pi + • • • + pk = n and U ^ tj for every i ^ j, we have 

K t = Sp Pl x • • • x Sp Pfe , 
It = (HS 2 U Pl © • • • © HS 2 W k ) © ImH . 
Decomposing v G H ra as v\ © • • • © v k with Vj G M. Pj , the Lie bracket in is 

(\(viiv'\ - v'^vl) - }-lmi(v*v')I pl \ 
[v,v% = ••. ©Im(uV) 

\ v k iv'* k - v' k iv* k - ±Im^V)/ pfc / 

If we consider, for 1 < j < k, the subalgebra h-,- of generated by M. Pj , 

f)j = U p i © (HS 2 U p i © ImH)) = (HP* © (HS%W* ffi ImH)) ffi R , 

we easily see that it is K t - invariant and only the factor Sp Pj of K t acts nontrivially on it. 
Since hj commutes with \)y for j ^ j', it follows that n t is the quotient, modulo a central 
ideal, of the product of the t)j. 

We conclude that (n t ,K t ) is a central reduction of the product of the pairs (f)j,Sp p ), 
where, in turn, each (fjj,Sp ) is the product of (H^ © (iJS^H^ © ImH),Sp P3 ) and the 
trivial pair (R,{1». 

10.4. The pairs (C 2n+1 © A 2 C 2n+1 , SU 2n+1 ) and (C 2n+1 © (A 2 C 2n+1 © R)), U 2 „+i). 

To fix the notation, we consider the second family of pairs, the other being analogous and 
simpler. The Lie bracket is 

(10.1) [v,v'} = hvV -v'%)®lm(v*v') . 

2 

For 

t = diag(iiJ Pl , . . . ,t k J Pk , 2q+ i) ffi u , 
with pi + • • • + p k + q = n and tj G R, tj 7^ tj 7^ for i 7^ j, we have 
#t = Sp Pl x • • • x Sp pfc x U 2(?+ i 
$ t = HS 2 U Pl J Pl ffi • • • ffi #S 2 H Pfc J Pk ffi A 2 C 2 " +1 ffi R . 
Like in the previous case, we split C 2n+1 as C 2pi ffi • • • ffi C 2 ' Pk ffi C 2g+1 , and set 
\]j — C 2pj ffi (HS 2 W j J Pj ffi R) , (j = l,...,Jfe) , hfe+i = C 2?+1 ffi (A 2 C 2?+1 ffi R) , 



i.e., the subalgebra generated by the j-th summand in C 2n+1 . Then, for 1 < j < k, 

t)j = U p J © (HS%W* © M 2 ) , 



and (f)j,Sp p ) is isomorphic to a central reduction of the pair in subsection 10.3 Finally, 
(xit,K t ) is isomorphic to a central reduction of the product of k pairs of this kind and the 
pair (fj fc+ i,U 2 , + i). 



10.5. The pair (C 2n © (A 2 C 



In 



su 2n ). 



The Lie bracket is given by (10.1 ). Any element z of 3 is conjugate, modulo an element of 
U 2n , to an element of the form 



(10.2) 



t = diag(*iJ w ,...,tfcJp fc ,02 ff ) ffi u 



with Pi + ■ ■ ■ + Pk + q = n and £j G K, tj 7^ tj for z 7^ j. 

If g > 0, then z and t are also conjugate under SU 2n . If q = 0, then there exists e t9 , unique 
up to a 2n-th root of unity, such that z is conjugate to 



t e = diag(tie ie J pi , t k e i6 J Pk 



u . 



Then 



K e ie t 



K, 



t ■ 



he iB t — e It 



For an element t as in (10.2), we have 
K t = Sp Pl x • • ■ x Sp Pfe x SU 2g 

HS 2 W^J Pl © • ■ ■ © HS 2 W k J Pk © A 2 C 29 



it 



HS 2 W^J, 



HS 2 W k J Pk ©zRdiag^Jj 



pi 5 



r 1 / ' 

' k °VkJ 



if q / , 
if g = . 



The discussion proceeds as in subsection 10.4 



10.6. The pairs (C 2 © C n © u 2 , U 2 x SU„), (C 2 © C 2 " © u 2 , U 2 x SpJ. 

Realizing the elements of as n x 2 (resp. 2n x 2) complex matrices, the Lie bracket is 

[U, t> J = -{V V — V V) . 

If t = diag(iti, it 2 ), then K t = K, n t = n if t\ = t 2 . 
If *i 7^2, we have 



(f)„©f)„,Ui x SU n x Ui) 

(f)2n © f)2n,Ui X Sp„ X Ui) 



where f) n = C n ffilR is the 2n + 1-dimensional Heisenberg algebra, the factor SU n , resp. Sp n , 
acts simultaneously on the two summands C n , resp. C 2n of D, and the two copies of Ui act 
independently each on one summand. 
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10.7. The pair (R 2 © O © (ImO © M), U 1 x Spin 7 ). 
Realizing t) as O 2 , the Lie bracket is 

[(«i, v 2 ), (v[, v' 2 )] = Im (v^ + v 2 v' 2 ) © Re (vtV 2 - v[v 2 ) . 

Take t — tii © a G a, where i is an imaginary unit in O. If t\ = 0, then K t = K and 
n t = n. 

If ti 7^ 0, then .fQ = SO2 x Spin 6 , where Spin 6 = SU4 consists of the elements k of Spin 7 
whose action on D commutes with left multiplication by i. Taking this as a complex structure 
on d, we can now realize D as C 4 © C 4 . So, n t = (C 4 © C 4 ) © a, with Lie bracket 

[{vi,v 2 ),(v' 1 ,v' 2 )] t = lm(viv[ +v 2 v' 2 ) ®Bb{v!v' 2 - v[v 2 ) , 

where conjugation and imaginary part are meant now in the complex sense. 

Also, S0 2 acts by two conjugate characters on the two subspaces V± = {(v, ±iv) : v G C 4 } 
of D, which are also SLVinvariant. Noticing that [t) + , D_] t = {0}, it is easy to verify that n t 
is the direct product 

n* = (tJ+ © R) © (t»_ © R) = fj 4 © f)4 • 
We can finally add an extra torus Ui to K t , acting on t) by scalar multiplication, without 
changing the orbits. In conclusion, 

(nt, K t ) = (f) 4 © fj 4) Ui x SU 4 x Ui) , 

where SU4 acts simultaneously on each factor and the two copies of Ui act independently 
each on one summand. 

10.8. The pair (H 2 © HP ©sp 2 , Sp 2 x Sp n ). 

Realizing the elements of D as n x 2 quaternionic matrices, the Lie bracket is 

[V, V \ = -{V V — V V) . 

If t = diag(iti, it 2 ), then K t = K, n t = n if t\ = t 2 = 0. The proper quotient pairs that 
appear in the other cases are as follows: 

(i) t\ = t 2 7^ 0: then K t = U 2 x Sp n and 3 t = u 2 , where U 2 , resp. u 2 , is embedded in 
Sp 2 , resp. in sp 2 , as the stabilizer of %l 2 . So {x\. t ,K t ) is one of the pairs at line 8 of 
Tables CD and [U 

(ii) h^t 2 = 0: then (n t , K t ) = ({M n © R) © (H n © ImH), \J 1 x Sp n x SpJ. 
(hi) ti ^ t 2 , t x t 2 ^ 0: then (f) 2n © h 2n , U x x Sp„ x Ui). 
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3 


Pi 






la 


so 2n 


K 2n 


S02n 


tr(z 2k ) 
(1 < k < n- 1) 

Pf 

Fl(2j 




%z 2k v 
(1 < fc < n- 1) 


1 K 


o(J2n+l 


m>2n + l 
IK 


•S02n+1 


tr(2 2fc ) 
(1 < fc < n) 




v^_^_^ — ^ J 
Pf(2|«) 


2 


u„ 


C" 




tr (; 2 ) fc ) 

(1 < fc < n) 


M 2 


v*(iz) k v 
(1 < fc < n-1) 


3 


Sp„ 


u n 




«1 , 22 , «3 , tr 2q 

(2 < fc < n) 


M 2 


U*2o« 

(1 < fc < n-1) 


4 


SU2n+l 


C 2n + 1 


A 2 € 2n + 1 


tr((zz) k ) 
(1 < fc < n) 


M a 


v*(zz) k v 
(1 < fc < n - 1) 


5 


U2n + 1 


C 2n + 1 


A 2 C 2n+1 ©1 


2, tr({z z ) k ) 
(*\ < fc < n) 


M 2 


U*(2 2o) fc « 

(^ < k < n) 


ft 
u 






A 2(p2n ffi to 


z,tr({z z ) k ) 
fl < fc < « — 1 ~) 

Pf(2) , Pf(2) 




v*(z z {) ) k v 
(1 < fc < n-1) 


7 


U 2 xSU„ 
(n > 2) 


c 2 ®c n 


U2 


tr (( i 2) fc ) 
(fc = l,2) 


tr ((tw ) ) 
(fc = l,2) 


itr (v*2«) 


8 


U 2 xS Pn 
(n > 2) 


c 2 ®c 2n 


U2 


tr {(iz) k ) 
(fc = l,2) 


tr({vv*) k ) 
(fc = l,2) 

W 2 M 2 -(^) 2 


itr (ti*2u) 


9 


Ui x Spin 7 


c®o 


ImOeK 


2o| 2 , 2 


M'H 2 - (Re («iv 2 )) 2 


Re (2 («iu 2 )) 


10a 


Sp 2 x Sp 1 


e 2 


Sp2 


tr(> 2fc ) 
(fc = l,2) 


M 2 


tr (zv(zv)*) 
tr ((2U1;* — i;u*2) 2 ) 


10b 


Sp 2 xSp„ 
(n > 2) 


e 2 ®e n 


Sp2 


tr(2 2fc ) 
(fc = l,2) 


tr((W) fc ) 
(fc = l,2) 


tr (zv(zv)*) 
tr ((2ut>* — vv* z) 2 ) 



Table 3. Systems of invariants on t) © 3 
(Legenda. Lines 3,5,6,9: £ 3o,<£ £ 3- Lines 7,8: find n — 1 at 1.2,7 resp.) 
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